blog

What can cognitive science tell us about art, and vice versa?

How do people make sense of Tuner Prize nominee Tino Sehgal’s These Associations? And what can cognitive scientists learn from the way they do it?

The result of the Turner prize 2013 has been reported worldwide as a shock win – mostly because this year, the chosen artwork is less shocking than usual.

French artist Laure Prouvost’s madcap films overturned both critical expectations and the bookie’s 6/1 odds against her to win. While William Hill and Ladbrokes had David Shrigley’s mischievous peeing sculptures as a 2/1 favourite, the critics had fancied Tino Sehgal’s live conceptual/performance artworks.

The Turner prize and its contestants have become famous for creating controversy and public discussion about the limits of what artists, galleries and critics consider worthy of aesthetic judgement. However, new research from Queen Mary University of London’s Cognitive Science Group suggests that audiences are generally unfazed by this kind of issue. In ordinary conversations between visitors to the Tate Modern, one of the most supposedly ‘experimental’ artworks in this year’s Turner Prize was immediately and unproblematically subjected to complex processes of aesthetic judgement by the viewing public.

To find out how (and if) people made sense of Tino Sehgal’s Turner Prize-nominated artwork These Associations, I recorded and analysed over two hundred ordinary conversations between visitors to the Tate Modern’s Turbine Hall.

lynpam1.1-vector

I collected recordings of visitors’ conversations over the duration of Sehgal’s performance piece for which the artist trained 300 participants (including the researcher himself) to engage in a series of coordinated movements on the floor of the 3400m² Turbine Hall.

Throughout gallery opening hours in the summer of 2012, up to 70 of these participants at a time would blend into the crowds of tourists, gallery-goers and school children that usually fill the hall. Sometimes Sehgal’s participants would engage visitors in one-to-one conversations, at others they would break out into songs or chants, run in a flocking pattern, or slow-walk through the hall as a large group. For the visitors on the balcony overlooking the hall, this was quite a spectacle, and they would often stand in couples or small groups talking and watching.

Although the But is it art? question is always in the headlines when the Turner Prize is announced, visitors to the Turbine Hall seemed not to care one way or the other. While the question was frequently invoked, and many guests simply assumed that what they were witnessing was an unauthorized and spontaneous ‘flashmob’, most conversations quickly moved on to discussing and describing the action unfolding in front of them—more like sports commentary or a nature documentary voice-over than art criticism.

People’s commentaries were often funny, insightful and playful. “Standing… Standing’s really contemporary right now” was a young American woman’s description of one of Sehgal’s living tableau scenes. “A bit like watching paint dry isn’t it” was one older English woman’s assessment, although she and her friend then discussed what they were observing in detail for half an hour. Several groups of children also learned to play ‘Pooh sticks’ with the piece: as Sehgal’s participants marched under the viewing bridge, they would pick favourites and then run to see whose would walk out first on the other side.

Even negative assessments of the work were then justified in discussion of the details of the piece: how it worked, what it looked like, who the trained participants were and how to tell them apart from ordinary gallery visitors, and what underlying rationale might account for different patterns, behaviours or movements.

Many visitors who arrived on the balcony talking to each other would lapse into long comfortable silences (quite unusual in normal conversation), while others would make ‘oohing’ and ‘aahing’ noises like people watching firework displays. However, both noisy and silent watchers would then explain their reactions to each other in terms of their analysis of the piece. Most striking was how people would seamlessly switch between talking about the artwork and talking about other aspects of their lives: work, music, London, the events of their day, etc., then back to the piece. Often assessments of the artwork were bound up in practical issues about whether to move on or stay watching, what to eat for lunch or what to view next.

The initial findings of this research suggest that seeing something as art—whether good or bad—is an ordinary, everyday social activity. Aesthetic judgements of Sehgal’s work did not come out as individual’s lofty ‘judgements of taste’, but were embedded in people’s everyday social activities. So the humour and skill with which people explained what they were experiencing to one another other was central to their enjoyment of Sehgal’s work – whether or not they categorized it as art.

The study of aesthetics in psychology, neuroscience and artificial intelligence has tended to concentrate on how people’s reactions to formal properties of traditional artistic objects or images, on survey data, or on tests of people’s basic perceptual or cognitive capabilities. These approaches tend to avoid dealing with artworks which—like many that are nominated for the Turner Prize – use non-conventional art forms because they may not be perceived ‘correctly’  as art outside of a gallery context.

But by looking at how people spontaneously explain their own perceptions of new and unfamiliar art forms to each other while in the process of experiencing them, this research explores how judgements of taste constantly adapt to changing social contexts. Finding out how interaction shapes the contexts in which aesthetic judgements ordinarily happen may be key to a more general understanding of how human cognition and perception adapt to constantly changing social situations and norms.

What can cognitive science tell us about art, and vice versa? Read More »

S.P.A.M.D.: a turn-analytical mnemonic

While trawling through the many piles of conversational data being thrown at me while I’m happily ensconced as a CLIC visiting graduate student at UCLA, I’ve made a little mnemonic device for myself to help me tackle a transcript turn by turn:

Turn # (lines)
– seq:
– pos:
– act:
– mrk:
– des:

For each turn of talk, I’m asking myself:

  1. Which turn number is it?
  2. Which lines does it occupy?
  3. What is it in its local sequence? (an FPP, an SPP etc.)
  4. Which position in the sequence does it occupy?
  5. What action does it implement? (If any.)
  6. What is it marked by? (If at all.)
  7. How is it designed/shaped?

Although there’s always lots more to ask of any turn, especially in terms of its use of conventional formulations, detailed lexical/syntactic/prosodic features and how the turns in or across sequences interrelate, this seems like a reasonable set of initial questions to ask when looking at a turn for the first time.

I’m OK with the acronym S.P.A.M.D. but would welcome any suggestions for other turn-analytic question labels beginning with M or even better – with E so I can complete the set.

S.P.A.M.D.: a turn-analytical mnemonic Read More »

Getting a footpedal working with Linux

UPDATE: I have subsequently found that transcribe works beautifully well and comes with some great pedal drivers. It’s a great piece of software and well worth the modest price. UPDATE 2: Please do not follow the instructions on this blog and expect them to work. It was written in 2013 so the software is *very* out of date and I doubt I’ll able to help you if it goes wrong.

John Heritage told me to go buy a foot pedal. I picked this Infinity USB-2 pedal up from ebay for $30.

416Zo0M9vXL._SX385_.jpg

It does make a tremendous difference not to have to constantly move one’s fingers from keyboard to mouse and back, although I had mitigated this considerably by using a thinkpad external keyboard with a trackpoint. However, this set-up did have some Rachmaninoff problems – requiring hand contortion to control audio, with attendant RSI/CTS-baiting effects that often resulted in sore wrists during transcription binges.

Getting the foot pedal working in Linux (Mint 14) was a minor mission, so I thought I’d document it for anyone else doing the same:

  1. First, I installed the lovely Footpedal GNOME integration control, which sadly seems a bit dormant.
  2. After installation, the script didn’t work immediately. I edited the script following Phillip Goodfellow’s instructions.

This involved:

a finding the script: # which footpedal —> /usr/bin/footpedal
b editing that script to comment out lines 119-126:

119 # Check whether your notification agent support
120 # icon-summary-body layout.
(...)
125 # self.reusable_notification.set_timeout(1000)
126 # Doesn't do anything

  1. Finally, after I got permissions issues (the error message suggested I run sudo chmod a+r /dev/usb/hiddev0 footpedal, which didn’t seem to work) I followed Jason Barnett’s and Richard Steffan’s advice and set up a udev rule:


# /etc/udev/rules.d/footpedal.rules
#
# Set permissions for USB footpedal Infinity IN-USB-1
#
# sudo lsusb -v reports
# Bus 002 Device 002: ID 05f3:00ff PI Engineering, Inc.
# Device Descriptor:
# <...>
# idVendor 0x05f3 PI Engineering, Inc.
# idProduct 0x00ff
# bcdDevice 1.20
# iManufacturer 1 VEC
# iProduct 2 VEC USB Footpedal
#
# Rules recommended by PLUG mailing list member, Jason Barnett on
September 20, 2011
#
# After changing rules issue this command: sudo service udev restart
#
SUBSYSTEM=="usb", SYSFS{idVendor}=="05f3", MODE="0666"
SUBSYSTEM=="usb", ATTRS{idVendor}=="05f3", MODE="0666"

One sudo service udev restart later, and the footpedal is up and running! Now to get it working with CLAN under windows emulation…

Getting a footpedal working with Linux Read More »

Journals that publish Conversation Analysis

A list of journals that repeatedly publish basic CA work rather than CA work that primarily applies CA to a domain-specific area of interest.

Many of the people who are doing interesting CA work today have published in one or more of these. Have I missed any important ones out? This list notably excludes work in medical sociology, classroom interaction and other disciplines in which CA is frequently applied.

And a much-too-small list of journals that look interesting for various reasons but have published less CA.

Update: here they are, listed by (possibly useless) impact factor ratings. Interesting though.

2006 2007 2008 2009 2010 2011 2012
PNAS 5.32 5.328 5.374 5.417 5.309 5.329 5.473
American Sociological Review 3.842 3.966 4.643 4.395 4.302 4.288 4.941
American Journal of Sociology 3.884 4.043 3.675 4.482 3.906 4.144 3.935
Research on Language and Social Interaction 1.079 1.51 0.827 0.515 2.566 2.092 2.023
Brain and Language 0.762 0.904 0.917 1.19 1.436 1.555 1.838
Human Communication Research 0.796 1.037 1.285 2.16 1.848 1.855 1.748
Journal of Linguistic Anthropology 0.434 1.454 0.884 0.283 1.089 0.995 1.591
Communication Monographs 0.655 1.149 0.971 0.951 1.087 0.75 1.543
PLoS One 0 1.016 1.691 1.896 2.001 1.762 1.512
Sociology 0.912 1.492 1.751 1.145 1.498 1.113 1.458
Social Psychology Quarterly 1.003 1.264 0.872 0.807 0.821 1.047 1.289
Journal of Communication 0.91 0.964 1.009 1.015 0.877 0.864 1.281
Qualitative Research 0.122 0.563 0.709 0.886 0.893 0.954 1.174
British Journal of Social Psychology 1.223 1.081 1.046 1.047 1.073 1.362 1.127
Language in Society 0.566 0.818 0.67 1.134 0.991 1.198 0.904
Discourse Studies 0.34 0.847 0.958 1.145 0.722 0.923 0.65
Discourse Processes 0.49 0.639 0.89 0.413 0.518 0.676 0.642
Journal of Pragmatics 0.348 0.371 0.687 0.542 0.574 0.676 0.638
Text and Talk 0.42 0.345 0.286 0.446 0.345 0.601 0.433
Language and Communication 0.445 0.351 0.754 0.441 0.778 0.565 0.425
Semiotica 0.242 0.26 0.255 0.225 0.215 0.231 0.205
Qualitative Research in Psychology 0 0.138 0.826 0.981 0.201 0.316 0.205

Journals that publish Conversation Analysis Read More »

Conversation analysis for geeks

Unfortunately Slideshare disabled audio annotation so you can’t hear this talk anymore – but you can read an updated blog post from 2015 that attempts something similar.

When presenting conversation analytic research in my Electronic Engineering and Computer Science department, or presenting at conferences in that context I tend to get some recurrent methodological questions which often preclude getting useful questions from people in other disciplines about the work I’m doing with those methods and the phenomena I’m actually interested in.

This is a problem because I’m one of the very few people using these kinds of methods in my department, and it would help to be able to find collaborators from other research areas and get quality feedback from people.

So, I took the opportunity of giving a short talk at Queen Mary’s EECS postgraduate student conference to try and hone a presentation of CA for geeks down to about 5-10 minutes, so if needs be, I can incorporate it into my talks to get through the methodological issues quickly but with enough substance to allow people to follow me into the meat of my research presentations.

 

Conversation analysis for geeks Read More »

Two forms of silent contemplation – abstract for ICCA 2014

Here’s the abstract of my upcoming presentation at ICCA 2014.

There’s a video of the talk with slides here.

Two forms of silent contemplation

  • Saul Albert (saul.albert@eecs.qmul.ac.uk)
  • Patrick GT Healey (ph@dcs.qmul.ac.uk)
  • 25/07/2013
Image from http://www.rodin.info/
Image from http://www.rodin.info/

Silent contemplation is often thought of as the canonical form of aesthetic appreciation, a process of solitary reflection during which the qualities of an artwork are apparently absorbed and considered. The ostensibly private, ineffable nature of such moments naturally suggests analysis in terms of individual neural, physiological or cognitive processes. However, one of the earliest achievements of conversation analysis was to show that silences can also be public conversational moves, used to achieve a variety of social actions.

This paper explores the structure of interactional silences in fragments of naturalistic conversation between people discussing artworks in galleries, at home or work, often in a “continuing state of incipient talk” (Schegloff and Sacks 1969). We distinguish between the sequential organisation of two common forms of contemplative silence: one pre-emptively prepared for by a speaker in advance of the silence, and one accounted for and in effect claimed as a contemplative silence by a speaker after the silence has occurred. Here we present an example of the latter form and outline the analytical context.

Sacks, Schegloff, and Jefferson (1974) introduce silence as a salient feature of conversation especially in relation to turn-taking. They distinguish between the ‘pause’ as an intra-turn silence and the ‘gap’ as a proper occasion for speaker transition, though they also show how these silences are ‘transformable’ in subsequent talk. If a potential gap opens up when a speaker stops talking, it may be transformed into a pause by that speaker resuming. Alternatively, if the gap is allowed to continue in silence, it may occasion a conversational ‘lapse’, which they illustrate with the following fragments in which pauses of 1 and 2 seconds are followed by lapses (indicated by arrows) of 16 and 14 seconds respectively.

(1)                             (C-J:2)

C: Well no I’ll drive (I don’ m//in’)
J: hhh
-> (1.0)
J: I meapt to offah.
--> (16.0)
J: Those shoes look nice when you keep putting stuff on 'em.
C: Yeah I 'ave to get another can cuz cuz it ran out

(...)

C: Yehhh=
J: =(ok) (2.0) I haven’t not. done anything the whole weekend.
C: (okay)
--> (14.0)
J: Dass a rilly nice swe::der,(.hh)'at's my favourite sweater 
   on you, it's the only one that looks right on you
C: mm huh.

Both lapses are terminated by J’s assessments of C’s clothes. These are hearable in their sequential positions as ‘first assessments’  (Pomerantz 1984), initiating a sequence of talk by projecting the relevance of some form of second assessment such as C’s minimal agreements “Yeah …” and “mm huh”. Although the first silence in fragment (1) is both started and ended by J, and the second starts with one speaker and ends with another, neither of J’s first assessments post-silence or C’s responses seem to orient to prior talk, suggesting that these silences are heard as discontinuous lapses ended with new topical sequences.

By contrast, in fragment (2), a conversation between Katherine, an art buyer and Stefan, a gallerist, ends in a 7.4 second silence. Katherine re-starts the conversation with a first assessment: “That is beautiful”.

(2)                             (BNC/KCV/003603)

KAT:    a- (.) an o̲l̲d̲er person rings and (1.3) you 
        sort of:f (1.7) tch .hhh (1.8) haff more 
        p̲a̲t̲ience wizz him or something. 
    (7.4) 
KAT:    So 
    (.7) 
KAT:    .hhh Ah I was .hh That is beautiful. 
    (.5)
STE:    Yes.

The initial assessments in both fragments (1) and (2) are not randomly inserted into the talk. Rather, as Mondada (2009) suggests in her analysis of food-talk in dinner conversations, such assessments are systematically used as a resource for relaunching conversation after a lapse or a troublesome spate of talk.

However, a closer look at fragment (2) demonstrates that while Katherine’s assessment accomplishes re-initiation of talk, prompting a minimal acknowledgement from Stefan, it does so in a way that seems to orient towards the preceding pre-lapse talk. Her initial “So”, suggests an incipient development of the prior topic [@Raymond2004], then her halting self-repair seems to be initiating a retrospective self-attributed account: “.hhh Ah I was .hh”. Finally Katherine produces “That is beautiful” hearable as both a first assessment, but possibly also as an account for the potentially troublesome 7.4 second lapse.

This post-lapse initial assessment can be seen to accomplish several actions simultaneously, including re-launching the conversation, but also asserting a retrospective claim to the prior lapse as a silence that is attributable and accounted for as contemplative.

This brief outline concentrates on only one way in which one form of contemplative silence is produced. Building on this analysis, we propose that silences in conversation can, in some circumstances, be shaped in different ways and to different extents for response (Stivers and Rossano 2012) and are treated by recipients as distinctively cognitive events in that ostensibly private internal processes are made manifest on the surface of the conversation.

A short review of other analytical approaches to conversational silence informs a discussion of whether contemplative silence may be seen as specific to contexts such as the joint viewing of artworks. We conclude that social practices of contemplation are generally available to participants in everyday conversation, and sketch out some of the implications for empirical and theoretical research into aesthetic response.

References

Mondada, Lorenza. 2009. “The methodical organization of talking and eating: Assessments in dinner conversations.” Food Quality and Preference 20 (dec): 558–571.

Pomerantz, A. 1984. “Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes.” In Structures of social action: Studies in Conversation Analysis, ed. J. Maxwell Atkinson and John Heritage, 57–102. Cambridge: Cambridge University Press.

Raymond, Geoffrey. 2004. “Prompting action: The stand-alone ’so’ in ordinary conversation.” Research on Language and Social Interaction 37: 185–218.

Sacks, H., E. A. Schegloff, and Gail Jefferson. 1974. “A simplest systematics for the organization of turn-taking for conversation.” Language 50: 696–735.

Schegloff, E. A., and Harvey Sacks. 1969. “Opening up closings.” Contract 49.

Stivers, Tanya, and Federico Rossano. 2012. “Mobilising response in interaction: a compositional view of questions.” In Questions: Formal, Functional and Interactional Perspectives, ed. Jan Peter de Ruiter, 70–90. Cambridge: Cambridge University Press.

Two forms of silent contemplation – abstract for ICCA 2014 Read More »

A continuing state of incipient talk

I was trying to find a way of describing the conversational situation I was recording last summer in which people were leaning on the rail of the first floor balcony of the Tate Modern, talking to each other while watching people below as they mill about, with some of them occasionally behaving in ways that might be considered performance art .

Then I remembered my supervisor Pat Healey giving me a paper on video-mediated-communication that used the phrase ‘a state of incipient talk’ and went looking for it, then got sucked into a rabbit hole tracking this formulation down through various bibliographical dependencies: it’s interesting.

“In consequence, the relatively delicate ways in which individuals subtly move from disengagement to engagement in face to face environments, especially when they are in a ‘state of incipient talk’, appear to be rendered problematic in video mediated co-presence.”

Heath, C., Luff, P., & Sellen, A. (1997). Reconfiguring media space: Supporting collaborative work. Video-mediated communication, 323–347.

Heath, Luff and Sellen use it to contrast the experience of co-present and remote office workers using EuroPARC’s Media Space video conferencing system. Oddly, although they put it in quote marks, they don’t attribute the phrase in their paper, and they don’t cite Schegloff and Sacks’ Opening Up Closings, in which, as far as I can tell, it first appears:

“What we are really dealing with is the problem of closing a conversation that ends a state of talk. It does not hold for members of a household in their living room, employees who share an office, passengers together in an automobile, etc., that is, persons who could be said to be in a ‘continuing state of incipient talk’. In such circumstances, there can be lapses of the operation of what we earlier called the basic features; for example, there can be silence after a speaker’s utterance which is neither an attributable silence nor a termination, which is seen as neither the suspension nor the violation of the basic features. These are adjournments, and seem to be done in a manner different from closings. Persons in such a continuing state of incipient talk need not begin new segments of conversation with exchanges of greetings, and need not close segments with closing sections and terminal exchanges. Much else would appear to be different in their conversational circumstances as compared to those in which a conversation is specifically ‘started up’, which we cannot detail here.”

Schegloff, E., & Sacks, H. (1973). Opening up closings. Semiotica, 8(4). pp. 289-327.

However, even in this paper, it appears in single quotes which makes me wonder if that’s just an artefact of the turn of phrase “could be said to be” used to introduce it, or whether it’s an unattributed quote from somewhere else.

Chuck Goodwin quotes a section of the above, and adds:

“Such talk differs from a single conversation in that it does not require exchanges of greetings or closings and permits extended lapses between talk.”

Goodwin, C. (1981). Conversational Organization: interaction between speakers and hearers. New York, London: Academic Press. pp. 23.

He devotes parts of chapter 3 of Conversational OrganisationNotes on the Organisation of Engagement” to “Talk within Disengagement” and “Accounting for Withdrawal” that I have to re-read with this question in mind.

Lorenza Mondada uses it to describe co-workers in adjoining desks in an open-plan workspace engaged in different foci, but occasionally formulating announcements and responses for each other prompted, as she has it, by silences following phone calls or other episodes of variously manifest solo engagement.

“This takes place in a situation of “continuing state of incipient talk” (GOFFMAN, 1963; SCHEGLOFF & SACKS, 1973), typical of open-plan workplaces, which favor various types of multi-activity (such as working while chatting with a colleague, helping somebody while continuing to work on the computer, etc.).”

Mondada, L. (2008). Using Video for a Sequential and Multimodal Analysis of Social Interaction, 9(3). pp. 27.

This is intriguing as she cites Ervin’s Goffman Behaviour in Public Spaces but gives no page reference and I can’t seem to find the phrase in there. However, Goffman is a compelling candidate for a source – as his chapter on unfocused interaction describes these kinds of situations, citing passengers on a  train and co-workers sharing an office:

“As might be expected, when the context firmly provides a dominant involvement that is outside the situation, as when riding in a train or airplane, then gazing out the window, or reverie, or sleeping may be quite permissible. In short, the more the setting guarantees that the participant has not withdrawn from what he ought to be involved in, the more liberty it seems he will have to manifest what would otherwise be considered withdrawal in the situation.”

Goffman, E. (1963). Behavior in public places. New York: The Free Press. pp. 58-9

However, this description frames the situation in terms of reverie or withdrawal, rather than the maintenance of a continuing state of incipient talk as such.

Goffman does talk about neighbouring office workers working to maintain “due disinvolvement” over long stretches of time with what he calls “subordinate involvements” with their work materials:

“This may even be carried to the point where one individual allows himself half-audible “progress grunts” such as, “What do you know!” “Hm hm,” “Let’s see,” without excusing himself to his co-worker.”

Goffman, ibid, pp.63.

But again, this seems to focus on the maintenance of accountable withdrawal rather than characterising the situation in more detail – although this is understandable given that Goffman’s analysis is framed by a situational and somewhat interpretative account of social propriety rather than Sacks and Schegloff’s empirical extrapolation into the detailed sequential orderliness of talk-in-interaction.

Finally Jeffrey D. Robinson quotes Schegloff quoting himself and Sacks in Opening up Closings in the relatively new Handbook of Conversation Aanalysis chapter on ‘Overall Structural Organisation’. I really like this quote because it suggests that these situations are particularly resistant to conversational unit segmentation by analysts, which seems to me (if conversationalists face the same challenges) to point to the importance of participants working to attribute and manage lapses in these situations. That’s definitely what I observe in the interactions I overhear at the Tate.

“Units or orders of organization of all sorts (or of only many sorts perhaps) can have — perhaps must have — both: a local organization, which operates via progressivity from one sub-unit to a next, at various levels of granularity; and an overall structural organization. The latter, of course, can only get its work done in the places provided by the former. The former — the local organization — can only get its emergent shaping by reference to the latter — or the several ‘latter’s which operate on it — for example, the overall structural organization of TCUs, of turns, of sequences, of courses of action or activities such as telling or answering, of the unit ‘a single conversation’, and of that sprawling marvel we call ‘a continuing state of incipient talk’ (Schegloff and Sacks, 1973 : 325 – 6), as though we understood it.”

Schegloff, E. A. (2011). Word repeats as unit ends. Discourse Studies, 13(3), 367–380., quoted in Robinson, J. (2012). Overall Structural Organization. The Handbook of Conversation Analysis, (i).

So – my search ended without a conclusion as to where that quote originally comes from, some clues point towards Goffman but I can’t find that phrasing or an unambiguously similar account of the situation. I think I’m going to email Schegloff and ask. In any case, if he thinks it’s a “sprawling marvel”, then I’m happy to use it to characterise the situation of standing in front of These Associations.

A continuing state of incipient talk Read More »

Identifying Emotions on the Basis of Manual Activation

Arash Eshghi started what turned out to be a very productive fight on our CogSci listserv with this press release: Carnegie Mellon Researchers Identify Emotions Based on Brain Activity and its attendant paper: Identifying Emotions on the Basis of Neural Activation.

I came up with a press release of my own, I might at some point get round to doing the Atlantic Salmon paper on the subject.


Press Release: University Researchers Identify Emotions Based on Finger Activity

New Study Extends “Palm Reading” Research to Feelings by Applying Machine Learning Techniques to Keyboard Data

For the first time, scientists at a university have identified which emotion a person is experiencing based on finger activity.

------------------

 :)      happy

 :(      sad

-----fig 1--------

The study combines keyboards and machine learning to measure finger signals to accurately read emotions in individuals. The findings illustrate how the finger categorizes feelings, giving researchers the first reliable methods to evaluate them.

“Our big breakthrough was the idea of testing typists, who are
experienced at expressing emotional states digitally. We were fortunate,
in that respect, that EECS has so many superb typists”

said a professor.

For the study, typists were shown the words for 9 emotions: anger, disgust, envy, fear, happiness, lust, pride, sadness and shame. and were recorded typing them multiple times in random order.

------------------
   :§      8(>_<)8
      8^O
            =)
x-(    ;-b...
 _ _ 
( " )   :-c
             :-@
    (-.-)

 DX        >:(
     :-S     
             ;-)
:0=     :-)

  :(     *:-}

  !-}       (-_-)
-----fig 2--------

The computer model, using statistical information to analyse keyboard activation patterns for 18 emotional words was able to guess the emotional content of photos being viewed using only the finger activity of the viewers.

“Despite manifest differences between people’s psychology, different
people tend to manually encode emotions in remarkably similar ways”

noted a graduate student.

A surprising finding from the research was that almost equivalent accuracy levels could be achieved even when the computer model made use of activation patterns in only one of a number of different subsections of the keyboard.

“This suggests that emotion signatures aren’t limited to specific
regions such as the qwerty parentheses cluster, but produce
characteristic patterns throughout a number of keyboard regions”

said a senior research programmer.


Identifying Emotions on the Basis of Manual Activation Read More »

3 recurrent complaints about conversation analysis

Explaining research that uses conversation analysis to people with strong leanings towards quantitative methods is hard. I find they often fixate on a few methodological issues and never get round to discussing the research itself.

Earlier this term I gave an intentionally provocative talk about conversation analysis (CA) and its use in Cognitive Science, to see what kinds of issues came out of the woodwork very strongly. It certainly worked – I got quite a vigorous and enjoyable smackdown from my colleagues, and although I wasn’t very well equipped to deal with it at the time, I got my wish of an enumerated set of these questions that fell into three broad categories.

I took these categories and developed a short presentation entitled “conversation analysis for geeks” that in future I will prepend to my presentations to try and deal with these issues before diving into my research material.

The issues outlined below are interesting ones though, and looking at them in a bit more detail has helped me to understand why it’s important to make the case for interactional analysis, rather than – as I’ve seen done very skilfully by some researchers – try to skirt or minimise these issues in presentations using rhetorically compelling, persuasive but methodologically unchallenging evidence.

Quantification

People in my research environment are very familiar with quantification and statistical significance as ways of explaining human action and cognition, almost to the point of prejudice about other forms of explanation. This is a well established problem, that both Emmanuel Schegloff and Paul ten Have have addressed at length. Unfortunately, I often have 15-20 minutes to present, and doubt I could get people to read these texts beforehand!

I’ve heard some researchers deal with this issue by invoking the word “qualitative” and “ethnography” to explain what they’re doing. However, I think this is an unsatisfying compromise. Calling what I’m doing qualitative ethnography makes it sound like I’m studying social groups and their cultures, whereas I’m studying the ethno-methods they use to articulate and negotiate their groupness and their environment. So this compromise feels unsatisfying in the context I’m working in because this misunderstanding gives rise to others – such as the common complaint that CA obsesses over obvious trivialities. These ethno-methods obviously would seem trivial if the object of study was the sociology of a specific culture as a whole, rather than the generalisable methods of how a culture is articulated through interaction.

I think it’s worth arguing this because there are actually compelling reasons for making strong claims about how these methods are generalisable, and working with quantitative methodologies to demonstrate that. For example, recent developments in CA such as Stivers et. al’s study of cultural variation in turn-taking use statistics very interestingly. Turn-taking, probably the most well-developed conversation analytic device, features as a motivating hypothesis for a statistical study of cultural variations. My sense is that this kind of CA-centric mixed methods collaboration is only possible because the empirical basis of CA is taken seriously as a systematic and formal analytical basis for the project.

Practical vs. theoretical reasoning

Or proof practices vs. logical proofs.

There are a few keywords that are real traps preventing interdisciplinary understanding. Although Cognitive Science is an interdisciplinary field, use of terms like ‘proof’ and ‘reasoning’ seem to trip my presentations up very easily. When I’ve presented sequences of talk as evidence of for analysis, I’ve had complaints that: “that’s evidence of interaction, not reasoning”. The problem seems to be that in Cognitive Science ‘reasoning’ necessarily implies a cognitive view of a mind as an abstract computing system, so ‘reasoning’ in this context is a private process, in the same way that ‘proof’ in Cognitive Science and a CS engineering context denotes formal, logical proof.

Again, not using this language at all, or trying to persuade people by avoiding the issue makes it hard to communicate the relevance of results to the interdisciplinary (but quantitatively/logically oriented) field of Cognitive Science. It may seem obvious that there are methodological differences and tensions between a cognitivist and phenomenological approach to studying human behaviour and action, but my experience of this research context is that methodological issues tend to be skirted for pragmatic, political and professional reasons.

Talking about ‘practical reasoning’ is the best way I’ve found to describe the way participants in conversation analyse the evidence available to them (whatever utterance or action occurred previously) to figure out what might be their relevant next action. Similarly, I’ve tried talking about ‘proof practices’ to describe the way participants in a conversation challenge and cross-check theirs and others’ social actions to establish the accuracy of their analysis. I’m not sure that these work – but again, it seems important to retain these terms to be able to make a claims about cognitive phenomena.

A good example of this is Celia Kitzinger’s article After post-cognitivism in which she writes about the convergence of conversation analysis and discursive psychology as interaction-analytical methods with something to say about cognitive science. As Stivers et. al. actually put into practice in their statistical paper, Kitzinger recommends using strong interactional evidence from CA to guide hypotheses that can form the basis of research with more quantitative methods – or in her case – cognitive models and simulations in computer and cognitive science.

Topical authority

This seems to me to be the strangest issue, perhaps because my training in aesthetics leads me to just assume that people have authority over their own expressions of likes and dislikes. I’m studying how people do these expressions – how they agree and disagree with each other over aesthetic assessments, but I’ve had people complain that if I want to find out about art, I should interview curators or experts instead of listening to what ‘ordinary people’ have to say about it.

I’m not quite sure what to make of this complaint – or how to react to it. It just seems obvious to me that what experts or people involved in aesthetic production say around their objects of professional aesthetic speculation and how they interact around them will be very different from the ways that people talk while walking around galleries or chatting about art and artists at home or at work. I’m interested in the latter – although I’m sure there’s a lot to be learned form both – and their intersections.

I think this question is informed again by a cognitivist theory of art that I recognise from Cognitive Science approaches to aesthetics, exemplified by this one from the introduction to “Aesthetic Science” by A. Shimamura.

science.png
From Shimamura, A. P., & Palmer, S. E. (Eds.). (2012). Aesthetic science: Connecting minds, brains, and experience. Oxford University Press, pp 24.

In this model of artistic communication, art is seen as an intentional state in the mind of the artist communicated to the beholder via the artwork: ie. art as (a certain kind of) thought. This amounts to a kind of transmission theory of art based on a familiar idea in communication theory – that a cognate concept is encoded in speech signals then decoded back into a concept in the mind of the hearer.

tumblr_m3lxwgf4Gc1rv5969o1_500.jpg

Informed by this understanding, it makes sense that if you want to find out about art, you go as close to the source – the mind of the artist – as possible, rather than the ‘decode’ side, where, presumably, the viewer or recipient has to contend with errors in ideation produced by the cognitive decoding process as well as any noise that may have been introduced to the signal.

I guess dealing with this issue is the entire subject of my PhD so might be quite a challenge to condense.

Other questions and misunderstandings

Speaking to Jon Hindmarsh at the fantastic summer institute he and his colleagues at King’s College run on video and the analysis of social interaction, he added the following from his experience of using and presenting qualitative video analysis in the context of sociology, anthropology and ethnography.

He said he gets complaints that:

  1. Video recordings skew people’s natural behaviour
  2. ‘Fly on the wall’ video recordings are ethically unsound

Perhaps as these fields are closer to the origins of these methodologies in sociology, these issues more familiar to researchers. I’ve not experienced these personally, probably because I haven’t used video in my analysis, and perhaps there’s an assumption that audio recordings will skew people’s behaviour less (not that I’ve noticed it in my recordings or the data I’ve collected from the Audio BNC). I have no idea why fly-on-the wall recordings might be ethically unsound if the data is treated carefully, but it sounds as though the issue may be more ideological than practical.

——-

I’m interested in hearing about other questions and misconceptions from other fields, and what strategies conversation analysis and ethnomethodology researchers are using to deal with them – or skirt around them in ways that don’t compromise their empirical claims.

3 recurrent complaints about conversation analysis Read More »

Use archive.org to park old websites without link rot

I’ve built dozens of websites over the years – both professionally and as part of artistic, research, teaching, or freelance projects. I’m still very proud of some of them and would like to show them to people and link to them. Other people have also linked to them extensively over the years, and those inbound links are useful.

The problem is that keeping all this stuff online takes maintenance and often causes headaches. I think I found a 10 second technical solution that I hope it doesn’t annoy the good people at archive.org.

Quick how-to:

  1. Find the best possible instance of your website on archive.org’s wayback machine.
  2. Create two redirect rule on your web server, one to block archive.org’s archiving script, and one to redirect all otehr traffic to archive.org

Breakdown:

Finding the best instance of your website

Archive.org indexes your website with varying regularity. You might want their latest version – but it doesn’t just depend on your website. Archive.org archives the environment of your site too.

There may be other websites that went offline before your site – if you link to the latest version that archive.org has, it may be that links out of your site are therefore dead or linking to domain squatters that moved in when people you linked to moved out.

The solution to this is to use archive.org’s time navigation bar to find the ‘optimum’ time at which your site, and it’s significant neighbours was in it’s heydey. Use this as the basis of the apache redirect rule.

Redirect rules

Most web servers allow you to define redirect rules. I use Apache. Apache provides you with several ways of doing this. You can either create redirect rules in your Apache configuration files via your sites-available, or add them to an .htaccess file in the root directory of your domain.

My redirect rule for my old art collective’s website looks like this:

<IfModule mod_rewrite.c>
        Options +FollowSymLinks
        RewriteEngine on

        # if it's archive.org trying to archive itself
        RewriteCond %{HTTP_USER_AGENT} ^ia_archiver
        RewriteRule ^.* - [F,L]

        # otherwise redirect to archive.org
        RewriteRule (.*) http://web.archive.org/web/20061205014515/http://twenteenthcentury.com/$1 [R=301,L]
</IfModule>

Thoughts?

I’d be interested to know what archive.org thinks of this use of their service. It seems such an obvious solution to a very widespread problem.

So many of my artist friends – particularly those who developed their own web skills for artistic purposes – now spend inordinate amounts of time keeping their websites alive across server and database incompatibilities, changes in the programming languages they used to create their services, and various other headaches.

This isn’t ideal – obviously the services can’t run on archive.org, and sadly, some of the more interesting bits of work I did were not very friendly to web crawlers so didn’t provide archive.org with much to go on, but that’s a good pointer for future work: make sure that your web projects are easily crawled by archive.org so you don’t have to sysadmin it for the rest of time.

Use archive.org to park old websites without link rot Read More »