conversational technology

Since 2007, our art collective The People Speak have been working on ways of trying to make the 13+ years of conversational oral history we have on archive public and searchable.

The conversations between people who meet around the Talkaoke table{{1}}, on street corners, at festivals, schools, or conferences have been recorded and archived on every format going from digi-beta to hi-8, RealMedia (oh God, the 90’s), and miniDV. For the last two years, we have finally moved to digital only, but the archival backlog is intimidating.

As challenging as the digitisation and archival issues are, the real problem is figuring out what people are talking about in this mountain of data. All the conversations facilitated by The People Speak are spontaneous, off the cuff, and open to people changing tack at any point. This has made it almost impossible to provide a thematically structured archive.

And this problem is not unique to this rather speciliased context. Aren’t all conversations, questions and answer sessions, and in fact, pretty much anything that involves people interacting with each other on video subject to the same contingencies of meaning?

If my early-stages training in Conversation Analysis have shown me anything, it’s that the apparent ‘content’ of a conversation is impossible to represent in any way other than through further conversations, and observations of how people work to repair their misunderstandings.

The Heckle System

The People Speak’s response to this problem has been the ‘Heckle’ system.

Using ‘Heckle’, an operator, or multiple participants in a conversation may search for and post google images, videos, web links, wikipedia articles or 140 characters of text, which then appear overlayed on a projected live video of the conversation.

Here is a picture of Heckle in use at the National Theatre, after a performance of Greenland.

As you can see, the people sitting around the Talkaoke table aren’t focused on the screens on which the camera view is projected live. The aim of the Heckle system is not to compete with the live conversation as such – but to be a backchannel, throwing up images, text and contextual explanations on the screen that enable new participants to understand what’s going on and join in the conversation.

The Heckle system also has a ‘cloud’ mode, in which it displays a linear representation of the entire conversation so far, including snapshots from the video at the moment that a heckle was created, alongside images, keywords, ‘chapter headings’ and video.

This representation of the conversation is often used as part of a rhetorical device by the Talkaoke host to review the conversation so far for the benefit of people who have just sat down to talk. A ‘Heckle operator’ can temporarily bring it up on a projection or other nearby display and the host then verbally summarises what has happened so far.

It also often functions as a modifier for what is being said. Someone is talking about a subject, and another participant or viewer posts an image which may contradict or ridicule their statement; someone notices and laughs, everyone’s attention is drawn to the screen momentarily, then returns to the conversation with this new interjection in mind. Some people use the Heckle system because they are too shy to take the microphone and speak. It may illustrate and reinforce or undermine and satirize. Some ‘heckles’ are made in reply to another heckle, some in reply to something said aloud, and vice versa.

If keywords are mentioned in the chat, those keywords can be matched to a timecode in the video, in effect, the heckled conversation becomes an index for the video recorded conversation: the conversation annotates the video{{2}}.

[[1]] Talkaoke, if you’ve never seen it before, is a pop-up talk-show invented by Mikey Weinkove of The People Speak in 1997. It involves a doughnut-shaped table, with a host sitting in the middle on a swivelly chair, passing the microphone around to anyone who comes and sits around the edge to talk. Check out the Talkaoke website if you’re curious.[[1]] [[2]] People don’t just post keywords. It’s quite important that they can post images and video too. The search terms they use to find these resources can also be recorded and used as keywords to annotate the video. A further possibility for annotation is that a corpus of pre-annotated images, such as those catalogued using the ESPgame could be used to annotate the video. This would then provide a second level of annotation: the annotations of the images used could be considered to be ‘nested’ annotations of the Talkaoke conversation. [[2]]