August 2011

I have three representations of Dr Who. S4E1 sitting in front of me:

1. A Semantic annotation of the episode based on the BBC Stories Ontology, both by Michael O. Jewell, Paul Rissen, and Toby Harris{{1}}.
2. The script for the episode, by Russel T Davies
3. A transcript of a couple of very rowdy screenings of the episode I organised at The People Speak HQ during which people heckled at the screen using short messages, images and video.

What’s hurting my brain at the moment is a question of representation. In this triple, if ‘represents’ is the predicate, which is the subject and which is the object?

• Is the Semantic annotation a representation of Dr Who S4E1: Partners in Crime the TV show, or is it a representation of the experience and interpretation of the person watching and annotating it? Or both?
• In the same way, is the transcript of the conversation a representation of people’s experience of watching the episode and making social sense of it together, but with a lot more context?
• Is the episode itself a representation of the shooting script?

Which philosophical texts can I turn to to help me make sense of this?

But most crucially (for my purposes), how can I best understand the similarities and differences between 1 (the semantic annotation) and 3 (the conversational transcript)?

I had a few ideas about this, mostly based on text-mining the conversation transcript via concept-extraction services such as LUpedia or Alchemy API to see if snatches of conversation can identify related entities within the annotation’s timeline, but feedback from the wonderful Jo Walsh was sceptical of this approach.

Basically, her critique was that

1. Using text-mining/concept extraction privileges text, whereas the heckle stream is very visual, and that seems important.
2. Entity-recognition/tagging services will yield a very variable quality of metadata. They’re designed to look for something specific in text and match it, and tend to require quite a bit of context (more than 140 characters of text)
3. Asking the question “to what extent can this be considered metadata” will get very inconclusive answers, which will question the point of asking the question in the first place.

I think I agree with point 3 – which questions the point of this blog post, but I think I still need some kind of bottom-up analysis of the relatedness of the data, and although I’d like to just disregard the slightly solipsistic question of what is representing what, it would be nice to be able to attribute any philosophical assertions to someone other than myself!

[[1]] Here’s the OWL for the episode. Here’s the n3 formatted annotation of the episode [[1]]

This scenario is designed to elicit and capture conversation between a group of people who are watching a specific episode of Dr. Who together.

The aim is to be able to compare existing formal metadata for this episode with this speculative ‘conversational metadata’, and evaluate it as an alternative representation of the same media object: Dr Who, Season 4, Episode 1, Partners in Crime.

The Setup

Two groups of eight people are invited to watch of an episode of Dr Who together on a large screen, during which they use their laptops and a simple text/image/video annotation interface to type short messages or send images onto the screen where they are visible as an overlay on top of the video of Dr Who.

The room is laid out in a ‘living room’ arrangement to support co-present viewing and interaction between participants, with comfortable seating arranged in a broad semi-circle, oriented towards a large projected video screen about ten feet away. Each participant is asked to bring their own laptop, tablet PC, or other wifi-enabled device with a web browser.

After making sure that all participants are on the network, there is an introductory briefing where they are given a presentation explaining the aims of the project and that they are free to walk around, use their laptops or just talk, and help themselves to food and drink during the screening.

The Annotation Tool

The system that the participants are using on their laptops/tablets or mobile phones has a simple web-based client, enabling viewers to choose a colour to identify themselves on the screen, and then type in 140 characters of text or search for images and video, before sending them to the main screen.

The Display Screen

The video of Dr Who is projected on a ‘main’ screen, alongside text, images and video clips sent by viewers in a fullscreen browser window. The images and videos sent by users have a coloured outline, and text-bubbles are coloured to indicate who posted them.

Images and videos run underneath the video in a ‘media bar’, while text bubbles posted by users drop onto the screen in random positions, but can be re-arranged on the screen or deleted by a ‘facilitator’.

Rationale

This ‘conversational scenario’ is a hybrid of various methods in which researchers have contrived situations to elicit data from participants. Before making any claims about the data gathered, some clarification of the purpose and methods of the scenario are necessary.

Ethnographic Studies of Social TV have tended to use audiovisual recordings of TV viewers in naturalistic settings as their primary source, and analytical methods such as Conversation Analysis and participant observation have been used to deepen their understanding of how people use existing TV devices and infrastructures in a social context.

HCI approaches to designing Social TV systems have built novel systems and undertaken user testing and competitive analysis of existing systems in order to better understand the relationship between people’s social behaviours around TV, and the heuristics of speculative Social TV{{1}} devices and services.

Semantic Web researchers have opportunistically found ways to ‘harvest’ and analyse communications activity from the Social Web, as well as new Social TV network services that track users’ TV viewing activity as a basis for content recommendations and social communication.

All of these approaches will be extremely useful in developing better conversational annotation systems, and improving understanding and design of Social TV for usability, and for making better recommendations.

Although the conversational scenario described borrows from each of these methods, it’s primary objective is to gather data from people’s mediated conversations had around a TV in order to build a case for seeing and using it as metadata.

System design, usability, viewer behaviour, user profiles, choices of video material, and the effect those issues have on the quality and nature of the captured metadata are a secondary concern to this first step in ascertaining whether conversations can be captured and treated as metadata pertaining to the video in the first place.

[[1]]I am using the term Social TV, following one of the earliest papers to coin the phrase by Oehlberg et. al (2006) to refer to Interactive TV systems that concentrate on the opportunities for viewer-to-viewer interaction afforded by the convergence of telecoms and broadcast infrastructures. Oehlberg, L., Ducheneaut, N., Thornton, J. D., Moore, R. J., & Nickell, E. (2006). Social TV: Designing for distributed, sociable television viewing. Proc. EuroITV (Vol. 2006, pp. 25–26). Retrieved from http://best.berkeley.edu/~lora/Publications/SocialTV_EuroITV06.pdf [[1]]