This paper contributes to research on the role of technology in ‘atypical’ interaction by examining a situation in which the technology takes on the stigma of atypicality. Building on our analysis, we argue that this approach provides a model for assistive technology research and development that moves away from a techno-medical model and focuses on how typicality (and atypicality) are achieved interactionally.
References
Alač, M., Gluzman, Y., Aflatoun, T., Bari, A., Jing, B., & Mozqueda, G. (2020). How Everyday Interactions with Digital Voice Assistants Resist a Return to the Individual. Evental Aesthetics, 9(1), 51.
Antaki, C., & Wilkinson, R. (2012). Conversation Analysis and the Study of Atypical Populations. In The Handbook of Conversation Analysis (pp. 533–550). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118325001.ch26
Barnes, S., & Bloch, S. (2020). Communication disorders, enchrony, and other-participation in repair. Clinical Linguistics & Phonetics, 34(10–11), 887–893. https://doi.org/10.1080/02699206.2020.1749886
Bedaf, S., Gelderblom, G. J., de Witte, L., Syrdal, D., Lehmann, H., Amirabdollahian, F., Dautenhahn, K., & Hewson, D. (2013). Selecting services for a service robot: Evaluating the problematic activities threatening the independence of elderly persons. 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), 1–6. https://doi.org/10.1109/ICORR.2013.6650458
Bottema-Beutel, K., Kapp, S. K., Lester, J. N., Sasson, N. J., & Hand, B. N. (2021). Avoiding Ableist Language: Suggestions for Autism Researchers. Autism in Adulthood, 3(1), 18–29. https://doi.org/10.1089/aut.2020.0014
Ekberg, K., Hickson, L., & Lind, C. (2020). Practices of Negotiating Responsibility for Troubles in Interaction Involving People with Hearing Impairment. In R. Wilkinson, J. P. Rae, & G. Rasmussen (Eds.), Atypical Interaction: The Impact of Communicative Impairments within Everyday Talk (pp. 409–433). Springer International Publishing. https://doi.org/10.1007/978-3-030-28799-3_14
García-Soler, Á., Facal, D., Díaz-Orueta, U., Pigini, L., Blasi, L., & Qiu, R. (2018). Inclusion of service robots in the daily lives of frail older users: A step-by-step definition procedure on users’ requirements. Archives of Gerontology and Geriatrics, 74, 191–196. https://doi.org/10.1016/j.archger.2017.10.024
Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human–Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278
Kendrick, K. H., & Drew, P. (2015). Recruitment: Offers, Requests, and the Organization of Assistance in Interaction. Research on Language & Social Interaction, 49(1), 1–19. https://doi.org/10.1080/08351813.2016.1126436
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018). Voice Interfaces in Everyday Life. Proceedings of the 2018 ACM Conference on Human Factors in Computing Systems (CHI’18). https://doi.org/10.1145/3173574.3174214
Robinson, J. D. (2006). Managing Trouble Responsibility and Relationships During Conversational Repair. Communication Monographs, 73(2), 137–161. https://doi.org/10.1080/03637750600581206
Sacks, H. (1984). On doing ‘being ordinary’. In J. Heritage & J. M. Atkinson (Eds.), Structures of social action: Studies in conversation analysis (pp. 413–429). Cambridge University Press.
Scherer, M. J. (2020). It is time for the biopsychosocialtech model. Disability and Rehabilitation: Assistive Technology, 15(4), 363–364. https://doi.org/10.1080/17483107.2020.1752319
Tuisku, O., Pekkarinen, S., Hennala, L., & Melkas, H. (2018). “Robots do not replace a nurse with a beating heart”: The publicity around a robotic innovation in elderly care. Information Technology & People, 32(1), 47–67. https://doi.org/10.1108/ITP-06-2018-0277
White, G. W., Lloyd Simpson, J., Gonda, C., Ravesloot, C., & Coble, Z. (2010). Moving from Independence to Interdependence: A Conceptual Model for Better Understanding Community Participation of Centers for Independent Living Consumers. Journal of Disability Policy Studies, 20(4), 233–240. https://doi.org/10.1177/1044207309350561
Wilkinson, R. (2019). Atypical Interaction: Conversation Analysis and Communicative Impairments. Research on Language and Social Interaction, 52(3), 281–299. https://doi.org/10.1080/08351813.2019.1631045
Wright, J. (2019). Robots vs migrants? Reconfiguring the future of Japanese institutional eldercare. Critical Asian Studies, 51(3), 331–354. https://doi.org/10.1080/14672715.2019.1612765
Wright, J. (2023). Robots won’t save Japan: An ethnography of eldercare automation. ILR Press, an imprint of Cornell University Press.
Policymakers are increasingly interested in using virtual assistants to augment social care services in the context of a demographic ageing crisis. At the same time, technology companies are market- ing conversational user interfaces (CUIs) and smart home systems as assistive technologies for elderly and disabled people. However, we know relatively little about how today’s commercially available CUIs are used to assist in everyday homecare activities, or how care service users and human care assistants interpret and adapt these technologies in practice. Here we report on a longitudinal conversation analytic case study to identify, describe, and share how CUIs can be used as assistive conversational agents in practice. The analysis reveals that, while CUIs can augment and support new capabilities in a homecare environment, they cannot replace the delicate interactional work of human care assistants. We ar- gue that CUI design is= best inspired and underpinned by a better understanding of the joint coordination of homecare activities
References
Alač, M., Gluzman, Y., Aflatoun, T., Bari, A., Jing, B., & Mozqueda, G. (2020). How Everyday Interactions with Digital Voice Assistants Resist a Return to the Individual. Evental Aesthetics, 9(1), 51.
Albert, S., & Hamann, M. (2021). Putting wake words to bed: We speak wake words with systematically varied prosody, but CUIs don’t listen. CUI 2021 – 3rd Conference on Conversational User Interfaces, 1–5. https://doi.org/10.1145/3469595.3469608
Amazon Echo. (2019). Amazon Alexa: Sharing is Caring. https://www.youtube.com/watch?v=225Wlg3pkdo
Archibald, M. M., & Barnard, A. (2018). Futurism in nursing: Technology, robotics and the fundamentals of care. Journal of Clinical Nursing, 27(11–12), 2473–2480. https://doi.org/10.1111/jocn.14081
Bedaf, S., Gelderblom, G. J., de Witte, L., Syrdal, D., Lehmann, H., Amirabdollahian, F., Dautenhahn, K., & Hewson, D. (2013). Selecting services for a service robot: Evaluating the problematic activities threatening the independence of elderly persons. 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), 1–6. https://doi.org/10.1109/ICORR.2013.6650458
Casey, D., Felzmann, H., Pegman, G., Kouroupetroglou, C., Murphy, K., Koumpis, A., & Whelan, S. (2016). What People with Dementia Want: Designing MARIO an Acceptable Robot Companion. In K. Miesenberger, C. Bühler, & P. Penaz (Eds.), Computers Helping People with Special Needs (pp. 318–325). Springer International Publishing. https://doi.org/10.1007/978-3-319-41264-1_44
Chappell, N. L., Dlitt, B. H., Hollander, M. J., Miller, J. A., & McWilliam, C. (2004). Comparative Costs of Home Care and Residential Care. The Gerontologist, 44(3), 389–400. https://doi.org/10.1093/geront/44.3.389
Dowling, S., Williams, V., Webb, J., Gall, M., & Worrall, D. (2019). Managing relational autonomy in interactions: People with intellectual disabilities. Journal of Applied Research in Intellectual Disabilities, 32(5), 1058–1066. https://doi.org/10.1111/jar.12595
García-Soler, Á., Facal, D., Díaz-Orueta, U., Pigini, L., Blasi, L., & Qiu, R. (2018). Inclusion of service robots in the daily lives of frail older users: A step-by-step definition procedure on users’ requirements. Archives of Gerontology and Geriatrics, 74, 191–196. https://doi.org/10.1016/j.archger.2017.10.024
Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522. https://doi.org/10.1016/S0378-2166(99)00096-X
Harmo, P., Taipalus, T., Knuuttila, J., Vallet, J., & Halme, A. (2005). Needs and solutions—Home automation and service robots for the elderly and disabled. 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 3201–3206. https://doi.org/10.1109/IROS.2005.1545387
House of Lords. (2021). Ageing: Science, Technology and Healthy Living (p. 132). House of Lords Science and Technology Select Committee. https://publications.parliament.uk/pa/ld5801/ldselect/ldsctech/183/183.pdf
Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human–Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278
Kendrick, K. H., & Drew, P. (2016). Recruitment: Offers, Requests, and the Organization of Assistance in Interaction. Research on Language and Social Interaction, 49(1), 1–19. https://doi.org/10.1080/08351813.2016.1126436
Kingston, A., Comas-Herrera, A., & Jagger, C. (2018). Forecasting the care needs of the older population in England over the next 20 years: Estimates from the Population Ageing and Care Simulation (PACSim) modelling study. The Lancet Public Health, 3(9), e447–e455. https://doi.org/10.1016/S2468-2667(18)30118-X
Krummheuer, A. L., Rehm, M., & Rodil, K. (2020). Triadic Human-Robot Interaction. Distributed Agency and Memory in Robot Assisted Interactions. Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 317–319. https://doi.org/10.1145/3371382.3378269
Levine, D. M., Ouchi, K., Blanchfield, B., Diamond, K., Licurse, A., Pu, C. T., & Schnipper, J. L. (2018). Hospital-Level Care at Home for Acutely Ill Adults: A Pilot Randomized Controlled Trial. Journal of General Internal Medicine, 33(5), 729–736. https://doi.org/10.1007/s11606-018-4307-z
Lipp, B. (2022). Caring for robots: How care comes to matter in human-machine interfacing. Social Studies of Science, 03063127221081446. https://doi.org/10.1177/03063127221081446
Maguire, D., Honeyman, M., Fenney, D., & Jabbal, J. (2021). Shaping the future of digital technology in health and social care. The King’s Fund. https://www.kingsfund.org.uk/publications/future-digital-technology-health-social-care
Sacks, H. (1984). On doing ‘being ordinary’. In J. Heritage & J. M. Atkinson (Eds.), Structures of social action: Studies in conversation analysis (pp. 413–429). Cambridge University Press.
Share, P., & Pender, J. (2018). Preparing for a Robot Future? Social Professions, Social Robotics and the Challenges Ahead. Irish Journal of Applied Social Studies, 18(1). https://doi.org/10.21427/D7472M
Stokoe, E., Sikveland, R. O., Albert, S., Hamann, M., & Housley, W. (2020). Can humans simulate talking like other humans? Comparing simulated clients to real customers in service inquiries. Discourse Studies, 22(1), 87–109. https://doi.org/10.1177/1461445619887537
Topol, E. (2019). The Topol Review: Preparing the healthcare workforce to deliver the digital future (p. 103). Health Education England. https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf
Tuisku, O., Pekkarinen, S., Hennala, L., & Melkas, H. (2018). “Robots do not replace a nurse with a beating heart”: The publicity around a robotic innovation in elderly care. Information Technology & People, 32(1), 47–67. https://doi.org/10.1108/ITP-06-2018-0277
White, G. W., Lloyd Simpson, J., Gonda, C., Ravesloot, C., & Coble, Z. (2010). Moving from Independence to Interdependence: A Conceptual Model for Better Understanding Community Participation of Centers for Independent Living Consumers. Journal of Disability Policy Studies, 20(4), 233–240. https://doi.org/10.1177/1044207309350561Post navigation
Wright, J. (2021). The Alexafication of Adult Social Care: Virtual Assistants and the Changing Role of Local Government in England. International Journal of Environmental Research and Public Health, 18(2), Article 2. https://doi.org/10.3390/ijerph18020812
Wright, J. (2023). Robots won’t save Japan: An ethnography of eldercare automation. ILR Press, an imprint of Cornell University Press.
The turn towards multimodality and embodiment in interaction research has yielded new terminology and representational schema in key publications (Nevile 2015). At the intersections between multidisciplinary fields, e.g., ethnomethodological and conversation analytic (EMCA) research exploring interactions between humans and ‘AI’, social robots, and conversational user interfaces, such methodological changes are even harder to track. How do these approaches to the meticulous, naturalistic study of technologies in (and of) social interaction reframe the key terms, schema and practices that constitute AI as a field of technosocial activity? Largely grounded in the EMCA Wiki bibliography, we map this emerging field and report on a bibliometric review of 90 publications directly relevant to EMCA studies of AI (broadly defined) including social robots and their components such as voice interfaces.
We found that the most works cited in the EMCA+AI corpus are classics from the canon of human interaction research (Garfinkel, Sacks, Schegloff, Goffman), including multimodality (Goodwin, Heath), human-machine interaction (Suchman), and STS (Latour). The most frequently cited texts are: Sacks, Schegloff and Jefferson’s (1974) ‘turn-taking paper’ (in 45% of items from the corpus), Garfinkel’s (1967) Studies (40%), and Suchman’s (1987) book (31%). Dealing specifically with AI from an EMCA perspective, Porcheron et al.’s 2018 paper on voice user interfaces is the most cited (11%). Apart from this one, two other texts feature as citation hubs: Alač’s (2016) and Pitsch et al.’s (2013) papers on social robots and embodiment. The study aims to provide a starting point for discussion about how concepts such as embodiment, agency and interaction are shared, used and understood through the practice of academic citation.
References
Nevile, M. (2015). The Embodied Turn in Research on Language and Social Interaction. Research on Language and Social Interaction, 48(2), 121–151.
Policymakers and care service providers are increasingly looking to technological developments in AI and robotics to augment or replace health and social care services in the context of a demographic ageing crisis (House of Lords, 2021; Kingston et al., 2018; Topol, 2019, pp. 54–55). However, there is still little evidence as to how these technologies might be applied to everyday social care situations (Maguire et al., 2021). This paper uses conversation analysis of ~100 hours of video recorded interactions between a disabled person, their virtual assistant (Alexa), and their (human) personal assistant to explore how routine care tasks are organized in a domestic setting. We focus on how the human participants organize conversational turn-space around ‘turns-at-use’ with the virtual assistant. Specifically, how turns-at-use ostensibly designed for the virtual assistant can recruit overhearing others. Further, we show how participants include the virtual assistant in their shared taskscape by, for example, putting ongoing activities and conversations on hold, visibly reorienting their bodies, or explicitly making themselves available for – or requesting – assistance when coordination trouble emerges between the machine-human dyad. Our findings show that virtual assistants expand the affordances of a homecare environment but do not replace the work of personal assistants.
References
Alač, M., Gluzman, Y., Aflatoun, T., Bari, A., Jing, B., & Mozqueda, G. (2020). How Everyday Interactions with Digital Voice Assistants Resist a Return to the Individual. Evental Aesthetics, 9(1), 51.
Amazon Echo. (2019). Amazon Alexa: Sharing is Caring. https://www.youtube.com/watch?v=225Wlg3pkdo
Archibald, M. M., & Barnard, A. (2018). Futurism in nursing: Technology, robotics and the fundamentals of care. Journal of Clinical Nursing, 27(11–12), 2473–2480. https://doi.org/10.1111/jocn.14081
Bedaf, S., Gelderblom, G. J., de Witte, L., Syrdal, D., Lehmann, H., Amirabdollahian, F., Dautenhahn, K., & Hewson, D. (2013). Selecting services for a service robot: Evaluating the problematic activities threatening the independence of elderly persons. 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), 1–6. https://doi.org/10.1109/ICORR.2013.6650458
Casey, D., Felzmann, H., Pegman, G., Kouroupetroglou, C., Murphy, K., Koumpis, A., & Whelan, S. (2016). What People with Dementia Want: Designing MARIO an Acceptable Robot Companion. In K. Miesenberger, C. Bühler, & P. Penaz (Eds.), Computers Helping People with Special Needs (pp. 318–325). Springer International Publishing. https://doi.org/10.1007/978-3-319-41264-1_44
Chappell, N. L., Dlitt, B. H., Hollander, M. J., Miller, J. A., & McWilliam, C. (2004). Comparative Costs of Home Care and Residential Care. The Gerontologist, 44(3), 389–400. https://doi.org/10.1093/geront/44.3.389
Dowling, S., Williams, V., Webb, J., Gall, M., & Worrall, D. (2019). Managing relational autonomy in interactions: People with intellectual disabilities. Journal of Applied Research in Intellectual Disabilities, 32(5), 1058–1066. https://doi.org/10.1111/jar.12595
García-Soler, Á., Facal, D., Díaz-Orueta, U., Pigini, L., Blasi, L., & Qiu, R. (2018). Inclusion of service robots in the daily lives of frail older users: A step-by-step definition procedure on users’ requirements. Archives of Gerontology and Geriatrics, 74, 191–196. https://doi.org/10.1016/j.archger.2017.10.024
Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522. https://doi.org/10.1016/S0378-2166(99)00096-X
Harmo, P., Taipalus, T., Knuuttila, J., Vallet, J., & Halme, A. (2005). Needs and solutions—Home automation and service robots for the elderly and disabled. 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 3201–3206. https://doi.org/10.1109/IROS.2005.1545387
House of Lords. (2021). Ageing: Science, Technology and Healthy Living (p. 132). House of Lords Science and Technology Select Committee. https://publications.parliament.uk/pa/ld5801/ldselect/ldsctech/183/183.pdf
Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human–Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278
Kendrick, K. H., & Drew, P. (2016). Recruitment: Offers, Requests, and the Organization of Assistance in Interaction. Research on Language and Social Interaction, 49(1), 1–19. https://doi.org/10.1080/08351813.2016.1126436
Kingston, A., Comas-Herrera, A., & Jagger, C. (2018). Forecasting the care needs of the older population in England over the next 20 years: Estimates from the Population Ageing and Care Simulation (PACSim) modelling study. The Lancet Public Health, 3(9), e447–e455. https://doi.org/10.1016/S2468-2667(18)30118-X
Krummheuer, A. L., Rehm, M., & Rodil, K. (2020). Triadic Human-Robot Interaction. Distributed Agency and Memory in Robot Assisted Interactions. Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 317–319. https://doi.org/10.1145/3371382.3378269
Levine, D. M., Ouchi, K., Blanchfield, B., Diamond, K., Licurse, A., Pu, C. T., & Schnipper, J. L. (2018). Hospital-Level Care at Home for Acutely Ill Adults: A Pilot Randomized Controlled Trial. Journal of General Internal Medicine, 33(5), 729–736. https://doi.org/10.1007/s11606-018-4307-z
Maguire, D., Honeyman, M., Fenney, D., & Jabbal, J. (2021). Shaping the future of digital technology in health and social care. The King’s Fund. https://www.kingsfund.org.uk/publications/future-digital-technology-health-social-care
Share, P., & Pender, J. (2018). Preparing for a Robot Future? Social Professions, Social Robotics and the Challenges Ahead. Irish Journal of Applied Social Studies, 18(1). https://doi.org/10.21427/D7472M
Stokoe, E., Sikveland, R. O., Albert, S., Hamann, M., & Housley, W. (2020). Can humans simulate talking like other humans? Comparing simulated clients to real customers in service inquiries. Discourse Studies, 22(1), 87–109. https://doi.org/10.1177/1461445619887537
Topol, E. (2019). The Topol Review: Preparing the healthcare workforce to deliver the digital future (p. 103). Health Education England. https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf
Tuisku, O., Pekkarinen, S., Hennala, L., & Melkas, H. (2018). “Robots do not replace a nurse with a beating heart”: The publicity around a robotic innovation in elderly care. Information Technology & People, 32(1), 47–67. https://doi.org/10.1108/ITP-06-2018-0277
White, G. W., Lloyd Simpson, J., Gonda, C., Ravesloot, C., & Coble, Z. (2010). Moving from Independence to Interdependence: A Conceptual Model for Better Understanding Community Participation of Centers for Independent Living Consumers. Journal of Disability Policy Studies, 20(4), 233–240. https://doi.org/10.1177/1044207309350561
In it, we argue (hopefully provocatively), that voice user interface designers should stop using wake words like “Alexa” and “Hey Siri” that are crowding each other out of the audible environment of the smart home. Our point is that, as interface elements, wake words are misleading for users who seem to treat them like fully-fledged interactional summons, when they’re really little more than glorified ‘on’ buttons.
We got a surprisingly positive response from the technically-inclined audience at the conference. I found it surprising mostly because wake words are so ubiquitous and central to the branding and functionality of today’s voice interfaces that it seems hard to imagine them being phased out in favour of something more prosaic.
Charles Goodwin. 2007. Interactive footing. In Reporting Talk, Elizabeth Holt and Rebecca Clift (eds.). Cambridge University Press, Cambridge, 16–46. DOI:https://doi.org/10.1017/CBO9780511486654.003
Alexa Hepburn and Galina B Bolden. 2017. Transcribing for social research. Sage, London.
William Housley, Saul Albert, and Elizabeth Stokoe. 2019. Natural Action Processing. In Proceedings of the Halfway to the Future Symposium 2019 (HTTF 2019), Association for Computing Machinery, Nottingham, United Kingdom, 1–4. DOI:https://doi.org/10.1145/3363384.3363478
Razan Jaber, Donald McMillan, Jordi Solsona Belenguer, and Barry Brown. 2019. Patterns of gaze in speech agent interaction. In Proceedings of the 1st International Conference on Conversational User Interfaces – CUI ’19, ACM Press, Dublin, Ireland, 1–10. DOI:https://doi.org/10.1145/3342775.3342791
Seung-Hee Lee. 2006. Second summonings in Korean telephone conversation openings. Language in Society. 35, 02. DOI:https://doi.org/10.1017/S0047404506060118
Gene H Lerner. 2003. Selecting next speaker: The context-sensitive operation of a context-free organization. Language in Society. 32, 02, 177–201. DOI:https://doi.org/10.1017/S004740450332202X
Ewa Luger and Abigail Sellen. 2016. “Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16), Association for Computing Machinery, New York, NY, USA, 5286–5297. DOI:https://doi.org/10.1145/2858036.2858288
Robert J. Moore and Raphael Arar. 2019. Conversational UX design: A practitioner’s guide to the natural conversation framework. Association for Computing Machinery, New York, NY, USA.
Clifford Nass and Youngme Moon. 2000. Machines and Mindlessness: Social Responses to Computers. Journal of Social Issues 56, 1 (2000), 81–103. DOI:https://doi.org/10.1111/0022-4537.00153
Hannah R. M. Pelikan and Mathias Broth. 2016. Why That Nao? In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems – CHI \textquotesingle16, ACM Press. DOI:https://doi.org/10.1145/2858036.2858478
Danielle Pillet-Shore. 2018. How to Begin. Research on Language and Social Interaction 51, 3 (July 2018), 213–231. DOI:https://doi.org/10.1080/08351813.2018.1485224
Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. In Proceedings of the 2018 ACM Conference on Human Factors in Computing Systems – CHI’18, ACM Press. DOI:https://doi.org/10.1145/3173574.3174214
Stuart Reeves, Martin Porcheron, and Joel Fischer. 2018. “This is not what we wanted”: designing for conversation with voice interfaces. Interactions 26, 1, 46–51. DOI:https://doi.org/10.1145/3296699
Harvey Sacks. 1995. Lectures on conversation. Wiley-Blackwell, London.
Emanuel A Schegloff. 1968. Sequencing in Conversational Openings. American Anthropologist 70, 6, 1075–1095. DOI:https://doi.org/10.1525/aa.1968.70.6.02a00030
Emanuel A Schegloff. 1988. Presequences and indirection: Applying speech act theory to ordinary conversation. Journal of Pragmatics 12, 1 (1988), 55–62.
Emanuel A Schegloff. 2007. Sequence organization in interaction: Volume 1: A primer in conversation analysis. Cambridge University Press, Cambridge.
There are very few – if any – software tools designed specifically for conversation analytic transcription, partly because so few conversation analysts use them, so there’s not really a ‘market’ for software developers to cater to.
Instead, we have to make do with tools that were designed for more generic research workflows, and which often build in analytic assumptions, constraints and visual metaphors that don’t necessarily correspond with EM/CA’s methodological priorities.
Nonetheless, most researchers that use digital transcription systems choose between two main paradigms.
the ‘list-of-turns’-type system represents interaction much like a Jeffersonian transcript: a rendering of turn-by-turn talk, line by line, laid out semi-diagrammatically so that lines of overlapping talk are vertically aligned on the page.
the ‘tiers-of-timelines’ system uses a horizontal scrolling timeline like a video editing interface, with multiple layers or ‘tiers’ representing e.g., each participant’s talk, embodied actions, and other types of action annotated over time.
CLAN’s interface (left) and ELAN’s (right) with transcripts of the same bit of audiovisual data
A key utility of both kinds of digital transcription systems is that they allow researchers to align media and transcript, and to use very precise timing tools to check the order and timing of their analytic observations.
For the most part, researchers choose lists-of-turns tools when their analysis is focused on conversation and audible turn-space, and tiers-of-timelines when their analysis focuses on video analysis of visible bodily action.
The problem for EM/CA researchers working with both these approaches, however, is that neither representational schema on its own, (nor any schema save whatever schema may have been constituted through the original interaction itself), is ideal for exploring and describing participants’ sense-making processes and resources.
Tiers-of-timelines representations are great for showing the temporal unfolding of simultaneous action, but it is hard to read more than a few seconds of activity at a glance. By contrast, lists-of-turns use the same basic schema as our well-practiced, mundane reading abilities to scan a page of text and take in the overall structure of a conversation, but reduce the fine-grained timing and multi-activity organization of complex embodied activities.
The reason I put this digital transcription workshop together was to combine existing, well-used software tools for digital transcription from both major paradigms, and to show how to work on a piece of data using both approaches. It’s not intended as a comprehensive ‘solution’, and there are many unresolved practical and conceptual issues, but I think it gives researchers the best chance to address their empirical concerns to help break away from the conceptual and disciplinary constraints that come from analyzing data using one, uniform type of user interface.
The workshop materials include slides (so people can use them to teach collaborators/students) as well as a series of short tutorial videos accompanying each practical exercise in the slides, along with some commentary from me.
My hope is that researchers will use and improve these materials, and possibly extend them to include additional tools (e.g., EXMARaLDA project tools, with which I’m less familiar). If you do, and you find ways to improve them with additional tips, hacks, or updated instructions that take into account new versions, please do let me know.
The abstract for a forthcoming article by myself and Dirk vom Lehn, soon to be liberated from the stalled pandemic year R&R cycle. Draft available if you’re willing to give feedback!
Abstract
While dance has often featured in sociological theory, there are relatively few empirical studies that explore the social practices through which people learn to dance together. This paper takes as its point of departure the way that partner dance is often featured as a metaphor to illustrate theories about social order and interaction. We examine a corpus of video data gathered as part of a day-long workshop and explore how novice dancers learn to perform some of the basic steps of a social dance in time with their partner and with the rhythmical environment. The analysis shows how dancers use rhythm, bodies, language and other resources to organize their social interactions and shows how ethnomethodology and conversation analysis provide a critical standpoint for examining sociological theories about the relationship between the body and the social.
I tried to make a mix between a film essay and a research presentation of work-in-progress, so it didn’t always work to put references on every slide. I’ve added them below with links to the data used where available.
Abstract
Sacks’ (1963) first published paper on ‘sociological description’ uses the metaphor of a mysterious ‘talking-and-doing’ machine, where researchers from different disciplines come up with incompatible, contradictory descriptions of its functionality. We may soon find ourselves in a similar situation to the one Sacks describes as AI continues to permeate the social sciences, and CA begins to encounter AI either as a research object, as a research tool, or more likely as a pervasive feature of both.
There is now a thriving industry in ‘Conversational AI’ and AI-based tools that claim to emulate or analyse talk, but both the study and use of AI within CA is still unusual. While a growing literature is using CA to study social robotics, voice interfaces, and conversational user experience design (Pelikan & Broth, 2016; Porcheron et al., 2018), few conversation analysts even use digital tools, let alone the statistical and computational methods that underpin conversational AI. Similarly, researchers and developers of conversational AI rarely cite CA research and have only recently become interested in CA as a possible solution to hard problems in natural language processing (NLP). This situation presents an opportunity for mutual engagement between conversational AI and CA (Housley et al., 2019). To prompt a debate on this issue, I will present three projects that combine AI and CA very differently and discusses the implications and possibilities for combined research programmes.
The first project uses a series of single case analyses to explore recordings in which an advanced conversational AI successfully makes appointments over the phone with a human call-taker. The second revisits debates on using automated speech recognition for CA transcription (Moore, 2015) in light of significant recent advances in AI-based speech-to-text, and includes a live demo of ‘Gailbot’, a Jeffersonian automated transcription system. The third project both uses and studies AI in an applied CA context. Using video analysis, it asks how a disabled man and his care worker interact while using AI-based voice interfaces and a co-designed ‘home automation’ system as part of a domestic routine of waking, eating, and personal care. Data are drawn from a corpus of ~500 hours of video data recorded by the participants using a voice-controlled, AI-based ‘smart security camera’ system.
These three examples of CA’s potential interpretations and uses of AI’s ‘talking-and-doing’ machines provide material for a debate about how CA research programmes might conceptualize AI, and use or combine it with CA in a mutually informative way.
Agre, P. (1997). Toward a critical technical practice: Lessons learned in trying to reform AI. Social Science, Technical Systems and Cooperative Work: Beyond the Great Divide. Erlbaum.
Alač, M., Gluzman, Y., Aflatoun, T., Bari, A., Jing, B., & Mozqueda, G. (2020). How Everyday Interactions with Digital Voice Assistants Resist a Return to the Individual. Evental Aesthetics, 9(1), 51.
Berger, I., Viney, R., & Rae, J. P. (2016). Do continuing states of incipient talk exist? Journal of Pragmatics, 91, 29–44. https://doi.org/10.1016/j.pragma.2015.10.009
Bolden, G. B. (2015). Transcribing as Research: “Manual” Transcription and Conversation Analysis. Research on Language and Social Interaction, 48(3), 276–280. https://doi.org/10.1080/08351813.2015.1058603
Brooker, P., Dutton, W., & Mair, M. (2019). The new ghosts in the machine: “Pragmatist” AI and the conceptual perils of anthropomorphic description. Ethnographic Studies, 16, 272–298. https://doi.org/10.5281/zenodo.3459327
Button, Graham. (1990). Going Up a Blind Alley: Conflating Conversation Analysis and Computational Modelling. In P. Luff, N. Gilbert, & D. Frolich (Eds.), Computers and Conversation (pp. 67–90). Academic Press. https://doi.org/10.1016/B978-0-08-050264-9.50009-9
Button, Graham, & Dourish, P. (1996). Technomethodology: Paradoxes and possibilities. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. http://dl.acm.org/citation.cfm?id=238394
Button, G., & Sharrock, W. (1996). Project work: The organisation of collaborative design and development in software engineering. Computer Supported Cooperative Work (CSCW), 5(4), 369–386. https://doi.org/10.1007/BF00136711
Duca, D. (2019). Who’s disrupting transcription in academia? — SAGE Ocean | Big Data, New Tech, Social Science. SAGE Ocean. https://ocean.sagepub.com/blog/whos-disrupting-transcription-in-academia
Fischer, J. E., Reeves, S., Porcheron, M., & Sikveland, R. O. (2019). Progressivity for voice interface design. Proceedings of the 1st International Conference on Conversational User Interfaces – CUI ’19, 1–8. https://doi.org/10.1145/3342775.3342788
Garfinkel, H. (1967). Studies in ethnomethodology. Prentice-Hall.
Goodwin, C. (1996). Transparent vision. In E. A. Schegloff & S. A. Thompson (Eds.), Interaction and Grammar (pp. 370–404). Cambridge University Press.
Heath, C., & Luff, P. (1992). Collaboration and control: Crisis management and multimedia technology in London Underground Line Control Rooms. Computer Supported Cooperative Work (CSCW), 1(1–2), 69–94.
Heritage, J. (1984). Garfinkel and ethnomethodology. Polity Press.
Heritage, J. (1988). Explanations as accounts: A conversation analytic perspective. In C. Antaki (Ed.), Analysing Everyday Explanation: A Casebook of Methods (pp. 127–144). Sage Publications.
Hoey, E. M. (2017). Lapse organization in interaction [PhD Thesis, Max Planck Institute for Psycholinguistics, Radbound University, Nijmegen]. http://bit.ly/hoey2017
Housley, W., Albert, S., & Stokoe, E. (2019). Natural Action Processing. In J. E. Fischer, S. Martindale, M. Porcheron, S. Reeves, & J. Spence (Eds.), Proceedings of the Halfway to the Future Symposium 2019 (pp. 1–4). Association for Computing Machinery. https://doi.org/10.1145/3363384.3363478
Kendrick, K. H. (2017). Using Conversation Analysis in the Lab. Research on Language and Social Interaction, 50(1), 1–11. https://doi.org/10.1080/08351813.2017.1267911
Leviathan, Y., & Matias, Y. (2018). Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone [Blog]. Google AI Blog. http://ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html
Local, J., & Walker, G. (2005). Methodological Imperatives for Investigating the Phonetic Organization and Phonological Structures of Spontaneous Speech. Phonetica, 62(2–4), 120–130. https://doi.org/10.1159/000090093
Luff, P., Gilbert, N., & Frolich, D. (Eds.). (1990). Computers and Conversation. Academic Press.
Moore, R. J. (2015). Automated Transcription and Conversation Analysis. Research on Language and Social Interaction, 48(3), 253–270. https://doi.org/10.1080/08351813.2015.1058600
Ogden, R. (2015). Data Always Invite Us to Listen Again: Arguments for Mixing Our Methods. Research on Language and Social Interaction, 48(3), 271–275. https://doi.org/10.1080/08351813.2015.1058601
O’Leary, D. E. (2019). Google’s Duplex: Pretending to be human. Intelligent Systems in Accounting, Finance and Management, 26(1), 46–53. https://doi.org/10.1002/isaf.1443
Pelikan, H. R. M., & Broth, M. (2016). Why That Nao? Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems – CHI \textquotesingle16. https://doi.org/10.1145/2858036.2858478
Pelikan, H. R. M., Broth, M., & Keevallik, L. (2020). “Are You Sad, Cozmo?”: How Humans Make Sense of a Home Robot’s Emotion Displays. Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 461–470. https://doi.org/10.1145/3319502.3374814
Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018). Voice Interfaces in Everyday Life. Proceedings of the 2018 ACM Conference on Human Factors in Computing Systems (CHI’18).
Reeves, S. (2017). Some conversational challenges of talking with machines. Talking with Conversational Agents in Collaborative Action, Workshop at the 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing. http://eprints.nottingham.ac.uk/40510/
Relieu, M., Sahin, M., & Francillon, A. (2019). Lenny the bot as a resource for sequential analysis: Exploring the treatment of Next Turn Repair Initiation in the beginnings of unsolicited calls. https://doi.org/10.18420/muc2019-ws-645
Robles, J. S., DiDomenico, S., & Raclaw, J. (2018). Doing being an ordinary technology and social media user. Language & Communication, 60, 150–167. https://doi.org/10.1016/j.langcom.2018.03.002
Sacks, H. (1984). On doing “being ordinary.” In J. Heritage & J. M. Atkinson (Eds.), Structures of social action: Studies in conversation analysis (pp. 413–429). Cambridge University Press.
Sacks, H. (1987). On the preferences for agreement and contiguity in sequences in conversation. In G Button & J. R. Lee (Eds.), Talk and social organization (pp. 54–69). Multilingual Matters.
Sacks, H. (1995a). Lectures on conversation: Vol. II (G. Jefferson, Ed.). Wiley-Blackwell.
Sacks, H., Schegloff, E. A., & Jefferson, G. (1974). A simplest systematics for the organization of turn-taking for conversation. Language, 50(4), 696–735. https://doi.org/10.2307/412243
Sahin, M., Relieu, M., & Francillon, A. (2017). Using chatbots against voice spam: Analyzing Lenny’s effectiveness. Proceedings of the Thirteenth Symposium on Usable Privacy and Security, 319–337.
Schegloff, E. A. (1988). On an Actual Virtual Servo-Mechanism for Guessing Bad News: A Single Case Conjecture. Social Problems, 35(4), 442–457. https://doi.org/10.2307/800596
Schegloff, E. A. (1993). Reflections on Quantification in the Study of Conversation. Research on Language & Social Interaction, 26(1), 99–128. https://doi.org/10.1207/s15327973rlsi2601_5
Schegloff, E. A. (2004). Answering the Phone. In G. H. Lerner (Ed.), Conversation Analysis: Studies from the First Generation (pp. 63–109). John Benjamins Publishing Company.
Schegloff, E. A. (2010). Some Other “Uh(m)s.” Discourse Processes, 47(2), 130–174. https://doi.org/10.1080/01638530903223380
Soltau, H., Saon, G., & Kingsbury, B. (2010). The IBM Attila speech recognition toolkit. 2010 IEEE Spoken Language Technology Workshop, 97–102. https://doi.org/10.1109/SLT.2010.5700829
Stivers, T. (2015). Coding Social Interaction: A Heretical Approach in Conversation Analysis? Research on Language and Social Interaction, 48(1), 1–19. https://doi.org/10.1080/08351813.2015.993837
Stokoe, E. (2011). Simulated Interaction and Communication Skills Training: The `Conversation-Analytic Role-Play Method’. In Applied Conversation Analysis (pp. 119–139). Palgrave Macmillan UK. https://doi.org/10.1057/9780230316874_7
Stokoe, E. (2013). The (In)Authenticity of Simulated Talk: Comparing Role-Played and Actual Interaction and the Implications for Communication Training. Research on Language & Social Interaction, 46(2), 165–185. https://doi.org/10.1080/08351813.2013.780341
Stokoe, E. (2014). The Conversation Analytic Role-play Method (CARM): A Method for Training Communication Skills as an Alternative to Simulated Role-play. Research on Language and Social Interaction, 47(3), 255–265. https://doi.org/10.1080/08351813.2014.925663
Stokoe, E., Sikveland, R. O., Albert, S., Hamann, M., & Housley, W. (2020). Can humans simulate talking like other humans? Comparing simulated clients to real customers in service inquiries. Discourse Studies, 22(1), 87–109. https://doi.org/10.1177/1461445619887537
Turing, A. (1950). Computing machinery and intelligence. Mind, 49, 433–460.
Walker, G. (2017). Pitch and the Projection of More Talk. Research on Language and Social Interaction, 50(2), 206–225. https://doi.org/10.1080/08351813.2017.1301310
Wong, J. C. (2019, May 29). “A white-collar sweatshop”: Google Assistant contractors allege wage theft. The Guardian. https://www.theguardian.com/technology/2019/may/28/a-white-collar-sweatshop-google-assistant-contractors-allege-wage-theft
I’ve been fascinated by a live camera stream showing a UK street since the start of the lockdown on the 23rd March 2020 because it’s shown how pedestrians interpret the 2m physical distancing rule.
I thought others might find it useful to read a quick how-to about grabbing video from live cameras – it’s a great way to get a quick and dirty bit of data to test a working hunch or do some rough analysis.
There are thousands of live cameras that stream to youtube, but it can be a bit cumbersome to capture more than a few seconds via more straightforward screen capture methods.
NB: before doing this for research purposes, check that doing so is compliant with relevant regional/institutional ethical guidelines.
Step 1: download and configure youtube-dl
Youtube-dl is a command line utility, which means you run it from the terminal window of your operating system of choice – it works fine on any Unix, on Windows or on Mac Os.
Don’t be intimidated if you’ve never used a command line before, you won’t have to do much beyond some copying and pasting.
I can’t do an installation how-to, but there are plenty online:
Mac:
https://www.youtube.com/watch?v=NhOkYXB2_QQ
Windows:
https://www.youtube.com/watch?v=xyPAaYq3H9E
I’ll assume that if you’re a Unix user, you know how to do this.
Step 2: copy and paste the video ID from the stream
Every Youtube video has a video ID that you can copy from the address bar of your browser. Here’s the one I used for the blog post mentioned above – which we’ve affectionately nicknamed the ‘kebab corpus’. The video ID is circled in red:
Step 3: use youtube-dl to begin gathering your video data
This bit is a little hacky – as in not really using the software as intended or documented, so I’ve created a short howto video. There might be better ways. If so, please let me know!
As I mention in that video – probably best not to leave youtube-dl running for too long on a stream as you might end up losing your video if something happens to interrupt the stream. I’ve captured up to half an hour at a time.
It’s possible to create scripts and automated actions for a variety of operating systems to do this all for you on a schedule – but if you need extensive video archives, I’d recommend contacting the owner of the stream to see if they can simply send you their high quality youtube archives.
We have a a fully funded PhD position available (deadline 6th March 2020) to work with myself, Prof. Charles Antaki and Prof. Liz Peel in collaboration with The Alzheimer’s Society to explore the opportunities, risks and wider issues surrounding the use of AI-based voice technologies such as the Amazon Echo and home automation systems in the lives of people with dementia.
Voice technologies are often marketed as enabling people’s independence. For example, Amazon’s “Sharing is Caring” advert for its AI-based voice assistant Alexa shows an elderly man being taught to use the ‘remind me’ function of an Amazon Echo smart speaker by his young carer. But how accessible are these technologies in practice? How are people with dementia and carers using them in creative ways to solve everyday access issues? And what are the implications for policy given the consent and privacy issues?
The project will combine micro and macro-levels of analysis and research. On the micro-level, the successful applicant will be trained and/or supported to use video analysis to study how people with dementia collaborate with their assistants to adapt and use voice technologies to solve everyday access issues. On the macro-level, the project will involve working on larger scale operations and policy issues with Ian Mcreath and Hannah Brayford at The Alzheimer’s Society and within the wider Dementia Choices Action Network (#DCAN).
Through this collaboration, the research will influence how new technologies are used, interpreted and integrated into personalised care planning across health, social care and voluntary, community and social enterprise sectors.
The deadline is the 6th March 2020 (see the job ad for application details). All you need to submit for a first round application is a CV and a short form, with a brief personal statement. We welcome applications from people from all backgrounds and levels of research experience (training in specific research methods will be provided where necessary). We especially welcome applications from people with first hand experience of disability and dementia, or with experience of working as a formal or informal carer/personal assistant.
This research will form part of the Adept at Adaptation project, looking at how disabled people adapt consumer AI-based voice technologies to support their independence across a wide range of impairment groups and applied settings.
