Month: August 2021

An artificial turn in social interaction research?

Jakub Mlynář, Andreas Liesenfeld, Lynn de Renata Topinková, Wyke Stommel, Lynn de Rijk, and Saul Albert for the 6th Copenhagen Multimodality Day: Interacting with AI

The turn towards multimodality and embodiment in interaction research has yielded new terminology and representational schema in key publications (Nevile 2015). At the intersections between multidisciplinary fields, e.g., ethnomethodological and conversation analytic (EMCA) research exploring interactions between humans and ‘AI’, social robots, and conversational user interfaces, such methodological changes are even harder to track. How do these approaches to the meticulous, naturalistic study of technologies in (and of) social interaction reframe the key terms, schema and practices that constitute AI as a field of technosocial activity? Largely grounded in the EMCA Wiki bibliography, we map this emerging field and report on a bibliometric review of 90 publications directly relevant to EMCA studies of AI (broadly defined) including social robots and their components such as voice interfaces.

We found that the most works cited in the EMCA+AI corpus are classics from the canon of human interaction research (Garfinkel, Sacks, Schegloff, Goffman), including multimodality (Goodwin, Heath), human-machine interaction (Suchman), and STS (Latour). The most frequently cited texts are: Sacks, Schegloff and Jefferson’s (1974) ‘turn-taking paper’ (in 45% of items from the corpus), Garfinkel’s (1967) Studies (40%), and Suchman’s (1987) book (31%). Dealing specifically with AI from an EMCA perspective, Porcheron et al.’s 2018 paper on voice user interfaces is the most cited (11%). Apart from this one, two other texts feature as citation hubs: Alač’s (2016) and Pitsch et al.’s (2013) papers on social robots and embodiment. The study aims to provide a starting point for discussion about how concepts such as embodiment, agency and interaction are shared, used and understood through the practice of academic citation.

References 

Nevile, M. (2015). The Embodied Turn in Research on Language and Social Interaction. Research on Language and Social Interaction, 48(2), 121–151.

The interactional coordination of virtual and personal assistants in a homecare setting

Saul Albert, Magnus Hamann & Elizabeth Stokoe (for the 6th Copenhagen Multimodality Day), October 2021.

Policymakers and care service providers are increasingly looking to technological developments in AI and robotics to augment or replace health and social care services in the context of a demographic ageing crisis (House of Lords, 2021; Kingston et al., 2018; Topol, 2019, pp. 54–55). However, there is still little evidence as to how these technologies might be applied to everyday social care situations (Maguire et al., 2021). This paper uses conversation analysis of ~100 hours of video recorded interactions between a disabled person, their virtual assistant (Alexa), and their (human) personal assistant to explore how routine care tasks are organized in a domestic setting. We focus on how the human participants organize conversational turn-space around ‘turns-at-use’ with the virtual assistant. Specifically, how turns-at-use ostensibly designed for the virtual assistant can recruit overhearing others. Further, we show how participants include the virtual assistant in their shared taskscape by, for example, putting ongoing activities and conversations on hold, visibly reorienting their bodies, or explicitly making themselves available for – or requesting – assistance when coordination trouble emerges between the machine-human dyad. Our findings show that virtual assistants expand the affordances of a homecare environment but do not replace the work of personal assistants.

References

House of Lords. (2021). Ageing: Science, Technology and Healthy Living (p. 132). House of Lords Science and Technology Select Committee. https://publications.parliament.uk/pa/ld5801/ldselect/ldsctech/183/183.pdf

Kingston, A., Comas-Herrera, A., & Jagger, C. (2018). Forecasting the care needs of the older population in England over the next 20 years: Estimates from the Population Ageing and Care Simulation (PACSim) modelling study. The Lancet Public Health3(9), e447–e455. https://doi.org/10.1016/S2468-2667(18)30118-X

Maguire, D., Honeyman, M., Fenney, D., & Jabbal, J. (2021). Shaping the future of digital technology in health and social care. The King’s Fund. https://www.kingsfund.org.uk/publications/future-digital-technology-health-social-care

Topol, E. (2019). The Topol Review: Preparing the healthcare workforce to deliver the digital future (p. 103). Health Education England. https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf

Putting wake words to bed

Magnus Hamann and I wrote a provocation paper for the third conference on Conversational User Interfaces 2021.

In it, we argue (hopefully provocatively), that voice user interface designers should stop using wake words like “Alexa” and “Hey Siri” that are crowding each other out of the audible environment of the smart home. Our point is that, as interface elements, wake words are misleading for users who seem to treat them like fully-fledged interactional summons, when they’re really little more than glorified ‘on’ buttons.

We got a surprisingly positive response from the technically-inclined audience at the conference. I found it surprising mostly because wake words are so ubiquitous and central to the branding and functionality of today’s voice interfaces that it seems hard to imagine them being phased out in favour of something more prosaic.

You can read the full paper on the ACM site, or a preprint here.

References

  1. Charles Goodwin. 2007. Interactive footing. In Reporting Talk, Elizabeth Holt and Rebecca Clift (eds.). Cambridge University Press, Cambridge, 16–46. DOI:https://doi.org/10.1017/CBO9780511486654.003
  2. Alexa Hepburn and Galina B Bolden. 2017. Transcribing for social research. Sage, London.
  3. William Housley, Saul Albert, and Elizabeth Stokoe. 2019. Natural Action Processing. In Proceedings of the Halfway to the Future Symposium 2019 (HTTF 2019), Association for Computing Machinery, Nottingham, United Kingdom, 1–4. DOI:https://doi.org/10.1145/3363384.3363478
  4. Razan Jaber, Donald McMillan, Jordi Solsona Belenguer, and Barry Brown. 2019. Patterns of gaze in speech agent interaction. In Proceedings of the 1st International Conference on Conversational User Interfaces – CUI ’19, ACM Press, Dublin, Ireland, 1–10. DOI:https://doi.org/10.1145/3342775.3342791
  5. Seung-Hee Lee. 2006. Second summonings in Korean telephone conversation openings. Language in Society. 35, 02. DOI:https://doi.org/10.1017/S0047404506060118
  6. Gene H Lerner. 2003. Selecting next speaker: The context-sensitive operation of a context-free organization. Language in Society. 32, 02, 177–201. DOI:https://doi.org/10.1017/S004740450332202X
  7. Ewa Luger and Abigail Sellen. 2016. “Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), Association for Computing Machinery, New York, NY, USA, 5286–5297. DOI:https://doi.org/10.1145/2858036.2858288
  8. Robert J. Moore and Raphael Arar. 2019. Conversational UX design: A practitioner’s guide to the natural conversation framework. Association for Computing Machinery, New York, NY, USA.
  9. Clifford Nass and Youngme Moon. 2000. Machines and Mindlessness: Social Responses to Computers. Journal of Social Issues 56, 1 (2000), 81–103. DOI:https://doi.org/10.1111/0022-4537.00153
  10. Hannah R. M. Pelikan and Mathias Broth. 2016. Why That Nao? In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems – CHI \textquotesingle16, ACM Press. DOI:https://doi.org/10.1145/2858036.2858478
  11. Danielle Pillet-Shore. 2018. How to Begin. Research on Language and Social Interaction 51, 3 (July 2018), 213–231. DOI:https://doi.org/10.1080/08351813.2018.1485224
  12. Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. In Proceedings of the 2018 ACM Conference on Human Factors in Computing Systems – CHI’18, ACM Press. DOI:https://doi.org/10.1145/3173574.3174214
  13. Stuart Reeves, Martin Porcheron, and Joel Fischer. 2018. “This is not what we wanted”: designing for conversation with voice interfaces. Interactions 26, 1, 46–51. DOI:https://doi.org/10.1145/3296699
  14. Harvey Sacks. 1995. Lectures on conversation. Wiley-Blackwell, London.
  15. Emanuel A Schegloff. 1968. Sequencing in Conversational Openings. American Anthropologist 70, 6, 1075–1095. DOI:https://doi.org/10.1525/aa.1968.70.6.02a00030
  16. Emanuel A Schegloff. 1988. Presequences and indirection: Applying speech act theory to ordinary conversation. Journal of Pragmatics 12, 1 (1988), 55–62.
  17. Emanuel A Schegloff. 2007. Sequence organization in interaction: Volume 1: A primer in conversation analysis. Cambridge University Press, Cambridge.