Saul Albert∗, Hendrik Buschmeier, Katharina Cyra, Christiane Even, Magnus Hamann, Jakub Mlynář, Hannah Pelikan, Martin Porcheron, Stuart Reeves, Philippe Sormani & Sylvaine Tuncer†

Citation: Albert, S., Buschmeier, H. Cyra, K., Even, C., Hamann, M., Licoppe, C., Mlynář, J., Pelikan, H., Porcheron, M., Reeves, S., Rudaz, D., Sormani, P., Tuncer, S. (2023, November 6-7). What ‘counts’ as an explanation in  social interaction? 2^nd TRR 318 Conference Measuring Understanding,  University of Paderborn, Paderborn, Germany.

Background

Measuring explainability in explainable AI (X-AI) usually involves technical methods for evaluating and auditing automated decision-making processes to highlight and eliminate potential sources of bias. By contrast, human practices of explaining usually involve doing explanation as a social action (Miller, 2019). X-AI’s transparent machine learning models can help to explain the proprietary ‘black boxes’ often used by high-stakes decision support systems in legal, financial, or diagnostic contexts (Rudin, 2019). However, as Rohlfing et al. (2021) point out, effective explanations (however technically accurate they may be), always involve processes of co-construction and mutual comprehension. Explanations usually involve at least two parties: the system and the user interacting with the system at a particular point in time, and ongoing contributions from both explainer and explainee are required. Without accommodating action, X-AI models appear to offer context-free, one-size-fits-all technical solutions that may not satisfy users’ expectations as to what constitutes a proper explanation.

What counts as an explanation?

If we accept that explanations are not simply stand-alone statements of causal relation, it can be hard to identify what should ‘count’ as an explanation in interaction (Ingram, Andrews, and Pitt, 2019). Research into explanation in ordinary human conversation has shown that explanations can be achieved through various practices tied to the local context of production cf. Schegloff, 1997. Moreover, explanations do not just appear anywhere in an interaction, but they are recurrently produced as responsive actions to fit an interactional ‘slot’ where someone has been called to account for something (Antaki, 1996). Sometimes explanations may also be produced as ‘initial’ moves in a sequence of action. In such cases, they are often designed to anticipate resistance and deal with. e.g., the routine contingencies that people cite when refusing to comply with an instruction (Antaki and Kent, 2012). Explanations as actions also perform and ‘talk into being’ social and institutional relationships such as doctor/patient, or teacher/student (Heritage and Clayman, 2010). Explainable AI systems, in this sense, become ‘accountable’ or ’transparent’ through their social uses (Button, 2003; Ehsan et al., 2021). We draw on concepts of explanation from Ethnomethodology and Conversation Analysis (Garfinkel, 1967; Garfinkel, 2002; Sacks, Schegloff, and Jefferson, 1974), Discursive Psychology (Edwards and Potter, 1992; Wiggins, 2016), and cognate fields like Distributed Cognition (Hutchins, 1995), and Enactivism (Di Paolo, Cuffari, and De Jaegher, 2018) to outline an empirical approach to explanation as a context-sensitive situated social practice (Suchman, 1987).

Explanations as joint actions Shared understanding is co-constructed through the achievement of coordinated social action (see e.g., Clark, 1996; Linell, 2009; Goodwin, 2017). Necessary and sufficient explanations cannot, therefore, be predefined by AI designers. Instead, explanation may be achieved through the achievement of joint actions with an AI in a specific context. If a system displays its capabilities in ways that match users’ expectations, they may achieve explanation (as contingently shared understanding) for all present intents and purposes. Explanation in this sense can never be considered complete – it could always be elaborated (cf. Garfinkel, 1967, pp. 73–75). While similar situations would involve predictability and regularity, this concept of explanation requires that participants jointly establish the relevant criteria and form for sufficient explanation with reference to the tasks and present purposes at hand, such as formulations of examples (Lee and Mlynář, 2023).

The self-explanatory nature of the social world Explanations are not only explicitly formulated, but are an inherent feature of the social world. Even without giving an explicit explanation, the design of an object provides ‘implicit’ explanations. Gibson (1979)’s concept of affordances, often used in system development, highlights that specific design features make specific actions relevant, e.g., a button that should be pressed, a lever that should be pulled (Norman, 1990). Situated social actions are also inherently recognisable (Levinson, 2013), even when mediated through AI such as in autonomous driving systems (Stayton, 2020). A slowly driving car will be recognised as not from the area (see, e.g., (Stayton, 2020)) and moving in certain ways can be recognised as, e.g., giving way to a pedestrian (Moore et al., 2019; Haddington and Rauniomaa, 2014). We could harness self-explanatory visibility (Nielsen, 1994) to design AI behaviours that are recognisable as specific social actions.

Miscommunication as explanation The practices of repair – the methods we use to recognise and deal with miscommunication (problems of speaking, hearing and understanding), as they occur in everyday interaction (Schegloff, Jefferson, and Sacks, 1977) – constitute pragmatic forms of explanation when they allow us to identify and resolve breakdowns of mutual understanding. For example, when someone says “huh?” in response to a ‘trouble source’ turn in spoken conversation, the speaker usually repeats the entire prior turn (Dingemanse, Torreira, and Enfield, 2013), whereas if the recipient had said “where?”, their response might only have solicited a repeat or reformulation of only the misheard place reference. These practices range from from tacit displays of uncertainty to explicit requests for clarification that solicit fully formed explanations and accounts (Raymond and Sidnell, 2019). These methods for real-time resolution of mutual (mis)understanding allows us, at least, to proceed with joint action in ways that establish and uphold explanation in action.

Explainability in action

This paper will use Conversation Analysis to examine episodes of Human-AI interactions, from a wide range of everyday interactional settings, and involving different technologies, user groups, and task orientations. Rather than attempting to establish a systematic or generalisable metric for explainability across interactional settings, the aim here is to encourage an extension of – and critical reflections on – our technical conceptualisation of explanation in X-AI.

References

Antaki, C. (1994). Explaining and Arguing: The Social Organization of Accounts. Sage.

Antaki, Charles (1996). “Explanation slots as resources in interaction”. In: British Journal of Social Psychology 35, pp. 415–432. doi: 10.1111/j.2044-8309.1996.tb01105.x.

Antaki, Charles and Alexandra Kent (2012). “Telling people what to do (and, sometimes, why): Contingency, entitlement and explanation in staff requests to adults with intellectual impairments”. In: Journal of Pragmatics 44, pp. 876–889. doi: 10.1016/j.pragma.2012.03.014.

Button, Graham (2003). “Studies of work in Human-Computer Interaction”. In: HCI Models, Theories, and Frameworks. Toward a Multidisciplinary Science. Ed. by John M. Carroll. San Francisco, CA, USA: Morgan Kaufmann, pp. 357–380. doi: 10.1016/b978-155860808-5/50013-7.

Clark, Herbert H. (1996). Using Language. Cambridge, UK: Cambridge University Press. doi: 10.1017/CBO9780511620539.

Depperman, A., & Haugh, M. (Eds.). (2021). Action ascription in interaction. Cambridge University Press.

Di Paolo, Ezequiel A., Elena Clare Cuffari, and Hanne De Jaegher (2018). Linguistic Bodies The Continuity between Life and Language. Cambridge, MA, USA: The MIT Press. doi: 10.7551/mitpress/11244.001.0001.

Dingemanse, Mark, Francisco Torreira, and Nick J. Enfield (2013). “Is “huh?” a universal word? Conversational infrastruc- ture and the convergent evolution of linugistic items”. In: PLoS ONE 8, e13636. doi: 10.1371/journal.pone.0078273.

Edwards, Derek and Jonathan Potter (1992). Discursive Psychology. London, UK: Sage.

Ehsan, Upol et al. (2021). “Expanding explainability: Towards social Transparency in AI systems”. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Yokohama, Japan. doi: 10.1145/3411764.3445188.

Garfinkel, Harold (1967). Studies in Ethnomethodology. Englewood Cliffs, NJ, USA: Prentice Hall.

Garfinkel, Harold (2002). Ethnomethodology’s program: Working out Durkheim’s aphorism. Lanham, Boulder, New York, Oxford: Rowman & Littlefield Publishers.

Garfinkel, H. (2021). Ethnomethodological Misreading of Aron Gurwitsch on the Phenomenal Field. Human Studies, 44(1), 19–42. https://doi.org/10.1007/s10746-020-09566-z

Gibson, James J. (1979). The Ecological Approach to Visual Perception. New York, NY, USA: Psychology Press. doi:10.4324/9781315740218.

Gill, V. T., & Maynard, D. W. (2006). Explaining illness: Patients’ proposals and physicians’ responses. In D. W. Maynard & J. Heritage (Eds.), Communication in Medical Care: Interaction between Primary Care Physicians and Patients (pp. 115–150). Cambridge University Press. https://doi.org/10.1017/CBO9780511607172.007

Goodwin, Charles (2017). Co-Operative Action. New York, NY, USA: Cambridge University Press. doi: 10 . 1017 / 9781139016735.

Haddington, Pentti and Mirka Rauniomaa (2014). “Interaction between road users”. In: Space and Culture 17, pp. 176–190. doi: 10.1177/1206331213508498.

Heller, V. (2016). Meanings at hand: Coordinating semiotic resources in explaining mathematical terms in classroom discourse. Classroom Discourse, 7(3), 253–275. https://doi.org/10.1080/19463014.2016.1207551

Heritage, J. (1988). Explanations as accounts: A conversation analytic perspective. In C. Antaki (Ed.), Analysing Everyday Explanation: A Casebook of Methods (pp. 127–144). Sage Publications.

Heritage, John and Steven Clayman (2010). Talk in Action. Interactions, Identities, and Institutions. Chichester, UK: Wiley. doi: 10.1002/9781444318135.

Hindmarsh, J., Reynolds, P., & Dunne, S. (2011). Exhibiting understanding: The body in apprenticeship. Journal of Pragmatics, 43(2), 489–503. https://doi.org/10.1016/j.pragma.2009.09.008

Hutchins, Edwins (1995). Cognition in The Wild. Cambridge, MA, USA: The MIT Press.

Ingram, J., Nick Andrews, and Andrea Pitt (2019). “When students offer explanations without the teacher explicitly asking them to”. In: Educational Studies in Mathematics 101, pp. 51–66. doi: 10.1007/s10649-018-9873-9.

Lee, Yeji and Jakub Mlynář (2023). ““For example” formulations and the interactional work of exemplification”. In: Human Studies. doi: 10.1007/s10746-023-09665-7.

Levinson, Stephen C. (2013). “Action formation and ascription”. In: The Handbook of Conversation Analysis. Ed. by Jack Sidnell and Tanya Stivers. Chichester, UK: Wiley-Blackwell, pp. 101–130. doi: https://doi.org/10.1002/ 9781118325001.ch6.

Linell, P (2009). Rethinking language, mind, and world dialogically: Interactional and contextual theories of human sense- making. Charlotte: Information Age Publishing inc.

Miller, Tim (2019). “Explanation in Artificial Intelligence: Insights from the social sciences”. In: Artificial Intelligence 267, pp.  1–38.  doi:  10.1016/j.artint.2018.07.007.

Moore, Dylan et al. (2019). “The Case for Implicit External Human-Machine Interfaces for Autonomous Vehicles”. In: Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. Utrecht, The Netherlands, pp. 295–307. doi: 10.1145/3342197.3345320.

Nielsen, Jakob (1994). “Enhancing the explanatory power of usability heuristics”. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Boston, MA, USA, pp. 152–158. doi: 10.1145/191666.191729.

Norman, Donald A. (1990). The Design of Everyday Things. New York, NY, USA: Basic Books.

Raymond, Geoffrey and Jack Sidnell (2019). “Interaction at the boundaries of a world known in common: initiating repair with “What do you nean?”” In: Research on Language and Social Interaction 52, pp. 177–192. doi: 10.1080/08351813. 2019.1608100.

Robinson, J. D. (2016). Accountability in Social Interaction. In J. D. Robinson (Ed.), Accountability in Social Interaction (pp. 1–44). Oxford University Press (OUP). https://doi.org/10.1093/acprof:oso/9780190210557.003.0001

Rohlfing, Katharina et al. (2021). “Explanation as a social practice: Toward a conceptual framework for the social design of AI systems”. In: IEEE Transactions on Cognitive and Developmental Systems 13, pp. 717–728. doi: 10.1109/TCDS. 2020.3044366.

Rudin, Cynthia (2019). “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead”. In: Nature Machine Intelligence 1, pp. 206–215. doi: 10.1038/s42256-019-0048-x.

Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson (1974). “A simplest systematics for the organization of turn-taking for conversation”. In: Language 50, pp. 696–735.

Schegloff, Emanuel A. (1997). “Whose Text? Whose Context?” In: Discourse & Society 8, pp. 165–187. doi: 10.1177/ 0957926597008002002.

Schegloff, Emanuel A., Gail Jefferson, and Harvey Sacks (1977). “The preference for self-correction in the organization of repair in conversation”. In: Language 53, pp. 361–382.

Sidnell, J. (2012). Declaratives, Questioning, Defeasibility. Research on Language & Social Interaction, 45(1), 53–60. https://doi.org/10.1080/08351813.2012.646686

Stayton, Erik Lee (2020). “Humanizing Autonomy: Social scientists’ and engineers’ futures for robotic cars”. PhD thesis.

Cambridge, MA, USA: Massachusetts Institute of Technology. doi: 1721.1/129050.

Suchman, Lucy A. (1987). Plans and Situated Actions: The Problem of Human-Machine Communication. Cambridge, UK: Cambridge University Press.

Wiggins, Sally (2016). Discursive Psychology. Theory, Method and Applications. Sage. doi: 10.4135/9781473983335.

∗Corresponding author: s.b.albert@lboro.ac.uk. Order of authors is alphabetical.

†SA, MH: Loughborough University; HB: Bielefeld University; KC: University of Duisburg-Essen; CE: Ruprecht Karl University of Heidelberg; JM:HES-SO Valais-Wallis; HP: Linköping University; MP: Swansea University; SR: University of Nottingham; PS: University of Lausanne; ST: King’s College

This paper contributes to research on the role of technology in ‘atypical’ interaction by examining a situation in which the technology takes on the stigma of atypicality. Building on our analysis, we argue that this approach provides a model for assistive technology research and development that moves away from a techno-medical model and focuses on how typicality (and atypicality) are achieved interactionally.

References

• Alač, M., Gluzman, Y., Aflatoun, T., Bari, A., Jing, B., & Mozqueda, G. (2020). How Everyday Interactions with Digital Voice Assistants Resist a Return to the Individual. Evental Aesthetics, 9(1), 51.
• Albert, S., Hamann, M., & Stokoe, E. (2023). Conversational User Interfaces in Smart Homecare Interactions: A Conversation Analytic Case Study. In ACM conference on Conversational User Interfaces (CUI ’23), 
• July 19–21, 2023, Eindhoven, Netherlands. ACM, New York, NY, USA 12 https://doi.org/10.1145/3571884.3597140
• Amazon Echo (Director). (2019). Amazon Echo & Alexa—Morning Ritual (60s). https://www.youtube.com/watch?v=rHsO-rXrLLo
• Amazon Echo (Director). (2019). Amazon Alexa: Sharing is Caring. https://www.youtube.com/watch?v=225Wlg3pkdo
• Antaki, C., & Wilkinson, R. (2012). Conversation Analysis and the Study of Atypical Populations. In The Handbook of Conversation Analysis (pp. 533–550). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781118325001.ch26
• Barnes, S., & Bloch, S. (2020). Communication disorders, enchrony, and other-participation in repair. Clinical Linguistics & Phonetics, 34(10–11), 887–893. https://doi.org/10.1080/02699206.2020.1749886
• Bedaf, S., Gelderblom, G. J., de Witte, L., Syrdal, D., Lehmann, H., Amirabdollahian, F., Dautenhahn, K., & Hewson, D. (2013). Selecting services for a service robot: Evaluating the problematic activities threatening the independence of elderly persons. 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), 1–6. https://doi.org/10.1109/ICORR.2013.6650458
• Bottema-Beutel, K., Kapp, S. K., Lester, J. N., Sasson, N. J., & Hand, B. N. (2021). Avoiding Ableist Language: Suggestions for Autism Researchers. Autism in Adulthood, 3(1), 18–29. https://doi.org/10.1089/aut.2020.0014
• Ekberg, K., Hickson, L., & Lind, C. (2020). Practices of Negotiating Responsibility for Troubles in Interaction Involving People with Hearing Impairment. In R. Wilkinson, J. P. Rae, & G. Rasmussen (Eds.), Atypical Interaction: The Impact of Communicative Impairments within Everyday Talk (pp. 409–433). Springer International Publishing. https://doi.org/10.1007/978-3-030-28799-3_14
• García-Soler, Á., Facal, D., Díaz-Orueta, U., Pigini, L., Blasi, L., & Qiu, R. (2018). Inclusion of service robots in the daily lives of frail older users: A step-by-step definition procedure on users’ requirements. Archives of Gerontology and Geriatrics, 74, 191–196. https://doi.org/10.1016/j.archger.2017.10.024
• Goodwin, C. (2007). Interactive footing. In E. Holt & R. Clift (Eds.), Reporting Talk (pp. 16–46). Cambridge University Press. https://doi.org/10.1017/CBO9780511486654.003
• Jackson, L., Haagaard, A., & Williams, R. (2022). Disability Dongle | Platypus. https://blog.castac.org/2022/04/disability-dongle/
• Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human–Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278
• Kendrick, K. H., & Drew, P. (2015). Recruitment: Offers, Requests, and the Organization of Assistance in Interaction. Research on Language & Social Interaction, 49(1), 1–19. https://doi.org/10.1080/08351813.2016.1126436
• Maguire, D., Honeyman, M., Fenney, D., & Jabbal, J. (2021). Shaping the future of digital technology in health and social care. The King’s Fund. https://www.kingsfund.org.uk/publications/future-digital-technology-health-social-care
• Porcheron, M., Fischer, J. E., Reeves, S., & Sharples, S. (2018). Voice Interfaces in Everyday Life. Proceedings of the 2018 ACM Conference on Human Factors in Computing Systems (CHI’18). https://doi.org/10.1145/3173574.3174214
• Robinson, J. D. (2006). Managing Trouble Responsibility and Relationships During Conversational Repair. Communication Monographs, 73(2), 137–161. https://doi.org/10.1080/03637750600581206
• Sacks, H. (1984). On doing ‘being ordinary’. In J. Heritage & J. M. Atkinson (Eds.), Structures of social action: Studies in conversation analysis (pp. 413–429). Cambridge University Press.
• Scherer, M. J. (2020). It is time for the biopsychosocialtech model. Disability and Rehabilitation: Assistive Technology, 15(4), 363–364. https://doi.org/10.1080/17483107.2020.1752319
• Tuisku, O., Pekkarinen, S., Hennala, L., & Melkas, H. (2018). “Robots do not replace a nurse with a beating heart”: The publicity around a robotic innovation in elderly care. Information Technology & People, 32(1), 47–67. https://doi.org/10.1108/ITP-06-2018-0277
• White, G. W., Lloyd Simpson, J., Gonda, C., Ravesloot, C., & Coble, Z. (2010). Moving from Independence to Interdependence: A Conceptual Model for Better Understanding Community Participation of Centers for Independent Living Consumers. Journal of Disability Policy Studies, 20(4), 233–240. https://doi.org/10.1177/1044207309350561
• Wilkinson, R. (2019). Atypical Interaction: Conversation Analysis and Communicative Impairments. Research on Language and Social Interaction, 52(3), 281–299. https://doi.org/10.1080/08351813.2019.1631045
• Wright, J. (2019). Robots vs migrants? Reconfiguring the future of Japanese institutional eldercare. Critical Asian Studies, 51(3), 331–354. https://doi.org/10.1080/14672715.2019.1612765
• Wright, J. (2023). Robots won’t save Japan: An ethnography of eldercare automation. ILR Press, an imprint of Cornell University Press.

Saul Albert, Magnus Hamann, Elizabeth Stokoe

Abstract:

Policymakers are increasingly interested in using virtual assistants to augment social care services in the context of a demographic ageing crisis. At the same time, technology companies are market- ing conversational user interfaces (CUIs) and smart home systems as assistive technologies for elderly and disabled people. However, we know relatively little about how today’s commercially available CUIs are used to assist in everyday homecare activities, or how care service users and human care assistants interpret and adapt these technologies in practice. Here we report on a longitudinal conversation analytic case study to identify, describe, and share how CUIs can be used as assistive conversational agents in practice. The analysis reveals that, while CUIs can augment and support new capabilities in a homecare environment, they cannot replace the delicate interactional work of human care assistants. We ar- gue that CUI design is= best inspired and underpinned by a better understanding of the joint coordination of homecare activities

References

Alač, M., Gluzman, Y., Aflatoun, T., Bari, A., Jing, B., & Mozqueda, G. (2020). How Everyday Interactions with Digital Voice Assistants Resist a Return to the Individual. Evental Aesthetics, 9(1), 51.

Albert, S., & Hamann, M. (2021). Putting wake words to bed: We speak wake words with systematically varied prosody, but CUIs don’t listen. CUI 2021 – 3rd Conference on Conversational User Interfaces, 1–5. https://doi.org/10.1145/3469595.3469608

Amazon Echo. (2019). Amazon Alexa: Sharing is Caring. https://www.youtube.com/watch?v=225Wlg3pkdo

Archibald, M. M., & Barnard, A. (2018). Futurism in nursing: Technology, robotics and the fundamentals of care. Journal of Clinical Nursing, 27(11–12), 2473–2480. https://doi.org/10.1111/jocn.14081

Bedaf, S., Gelderblom, G. J., de Witte, L., Syrdal, D., Lehmann, H., Amirabdollahian, F., Dautenhahn, K., & Hewson, D. (2013). Selecting services for a service robot: Evaluating the problematic activities threatening the independence of elderly persons. 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR), 1–6. https://doi.org/10.1109/ICORR.2013.6650458

Casey, D., Felzmann, H., Pegman, G., Kouroupetroglou, C., Murphy, K., Koumpis, A., & Whelan, S. (2016). What People with Dementia Want: Designing MARIO an Acceptable Robot Companion. In K. Miesenberger, C. Bühler, & P. Penaz (Eds.), Computers Helping People with Special Needs (pp. 318–325). Springer International Publishing. https://doi.org/10.1007/978-3-319-41264-1_44

Chappell, N. L., Dlitt, B. H., Hollander, M. J., Miller, J. A., & McWilliam, C. (2004). Comparative Costs of Home Care and Residential Care. The Gerontologist, 44(3), 389–400. https://doi.org/10.1093/geront/44.3.389

Dowling, S., Williams, V., Webb, J., Gall, M., & Worrall, D. (2019). Managing relational autonomy in interactions: People with intellectual disabilities. Journal of Applied Research in Intellectual Disabilities, 32(5), 1058–1066. https://doi.org/10.1111/jar.12595

García-Soler, Á., Facal, D., Díaz-Orueta, U., Pigini, L., Blasi, L., & Qiu, R. (2018). Inclusion of service robots in the daily lives of frail older users: A step-by-step definition procedure on users’ requirements. Archives of Gerontology and Geriatrics, 74, 191–196. https://doi.org/10.1016/j.archger.2017.10.024

Goodwin, C. (2000). Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10), 1489–1522. https://doi.org/10.1016/S0378-2166(99)00096-X

Harmo, P., Taipalus, T., Knuuttila, J., Vallet, J., & Halme, A. (2005). Needs and solutions—Home automation and service robots for the elderly and disabled. 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 3201–3206. https://doi.org/10.1109/IROS.2005.1545387

House of Lords. (2021). Ageing: Science, Technology and Healthy Living (p. 132). House of Lords Science and Technology Select Committee. https://publications.parliament.uk/pa/ld5801/ldselect/ldsctech/183/183.pdf

Jackson, L., Haagaard, A., & Williams, R. (2022). Disability Dongle | Platypus. https://blog.castac.org/2022/04/disability-dongle/

Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014). Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review. International Journal of Human–Computer Interaction, 30(5), 369–393. https://doi.org/10.1080/10447318.2013.873278

Kendrick, K. H., & Drew, P. (2016). Recruitment: Offers, Requests, and the Organization of Assistance in Interaction. Research on Language and Social Interaction, 49(1), 1–19. https://doi.org/10.1080/08351813.2016.1126436

Kingston, A., Comas-Herrera, A., & Jagger, C. (2018). Forecasting the care needs of the older population in England over the next 20 years: Estimates from the Population Ageing and Care Simulation (PACSim) modelling study. The Lancet Public Health, 3(9), e447–e455. https://doi.org/10.1016/S2468-2667(18)30118-X

Krummheuer, A. L., Rehm, M., & Rodil, K. (2020). Triadic Human-Robot Interaction. Distributed Agency and Memory in Robot Assisted Interactions. Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, 317–319. https://doi.org/10.1145/3371382.3378269

Levine, D. M., Ouchi, K., Blanchfield, B., Diamond, K., Licurse, A., Pu, C. T., & Schnipper, J. L. (2018). Hospital-Level Care at Home for Acutely Ill Adults: A Pilot Randomized Controlled Trial. Journal of General Internal Medicine, 33(5), 729–736. https://doi.org/10.1007/s11606-018-4307-z

Lipp, B. (2022). Caring for robots: How care comes to matter in human-machine interfacing. Social Studies of Science, 03063127221081446. https://doi.org/10.1177/03063127221081446

Maguire, D., Honeyman, M., Fenney, D., & Jabbal, J. (2021). Shaping the future of digital technology in health and social care. The King’s Fund. https://www.kingsfund.org.uk/publications/future-digital-technology-health-social-care

Sacks, H. (1984). On doing ‘being ordinary’. In J. Heritage & J. M. Atkinson (Eds.), Structures of social action: Studies in conversation analysis (pp. 413–429). Cambridge University Press.

Share, P., & Pender, J. (2018). Preparing for a Robot Future? Social Professions, Social Robotics and the Challenges Ahead. Irish Journal of Applied Social Studies, 18(1). https://doi.org/10.21427/D7472M

Stokoe, E., Sikveland, R. O., Albert, S., Hamann, M., & Housley, W. (2020). Can humans simulate talking like other humans? Comparing simulated clients to real customers in service inquiries. Discourse Studies, 22(1), 87–109. https://doi.org/10.1177/1461445619887537

Topol, E. (2019). The Topol Review: Preparing the healthcare workforce to deliver the digital future (p. 103). Health Education England. https://topol.hee.nhs.uk/wp-content/uploads/HEE-Topol-Review-2019.pdf

Tuisku, O., Pekkarinen, S., Hennala, L., & Melkas, H. (2018). “Robots do not replace a nurse with a beating heart”: The publicity around a robotic innovation in elderly care. Information Technology & People, 32(1), 47–67. https://doi.org/10.1108/ITP-06-2018-0277

White, G. W., Lloyd Simpson, J., Gonda, C., Ravesloot, C., & Coble, Z. (2010). Moving from Independence to Interdependence: A Conceptual Model for Better Understanding Community Participation of Centers for Independent Living Consumers. Journal of Disability Policy Studies, 20(4), 233–240. https://doi.org/10.1177/1044207309350561Post navigation

Wright, J. (2021). The Alexafication of Adult Social Care: Virtual Assistants and the Changing Role of Local Government in England. International Journal of Environmental Research and Public Health, 18(2), Article 2. https://doi.org/10.3390/ijerph18020812

Wright, J. (2023). Robots won’t save Japan: An ethnography of eldercare automation. ILR Press, an imprint of Cornell University Press.

Saul Albert, Magnus Hamann & Elizabeth Stokoe (for the 6th Copenhagen Multimodality Day), October 2021.

Abstract

Policymakers and care service providers are increasingly looking to technological developments in AI and robotics to augment or replace health and social care services in the context of a demographic ageing crisis (House of Lords, 2021; Kingston et al., 2018; Topol, 2019, pp. 54–55). However, there is still little evidence as to how these technologies might be applied to everyday social care situations (Maguire et al., 2021). This paper uses conversation analysis of ~100 hours of video recorded interactions between a disabled person, their virtual assistant (Alexa), and their (human) personal assistant to explore how routine care tasks are organized in a domestic setting. We focus on how the human participants organize conversational turn-space around ‘turns-at-use’ with the virtual assistant. Specifically, how turns-at-use ostensibly designed for the virtual assistant can recruit overhearing others. Further, we show how participants include the virtual assistant in their shared taskscape by, for example, putting ongoing activities and conversations on hold, visibly reorienting their bodies, or explicitly making themselves available for – or requesting – assistance when coordination trouble emerges between the machine-human dyad. Our findings show that virtual assistants expand the affordances of a homecare environment but do not replace the work of personal assistants.

Magnus Hamann and I wrote a provocation paper for the third conference on Conversational User Interfaces 2021.

In it, we argue (hopefully provocatively), that voice user interface designers should stop using wake words like “Alexa” and “Hey Siri” that are crowding each other out of the audible environment of the smart home. Our point is that, as interface elements, wake words are misleading for users who seem to treat them like fully-fledged interactional summons, when they’re really little more than glorified ‘on’ buttons.

We got a surprisingly positive response from the technically-inclined audience at the conference. I found it surprising mostly because wake words are so ubiquitous and central to the branding and functionality of today’s voice interfaces that it seems hard to imagine them being phased out in favour of something more prosaic.

You can read the full paper on the ACM site, or a preprint here.

I’ve been fascinated by a live camera stream showing a UK street since the start of the lockdown on the 23rd March 2020 because it’s shown how pedestrians interpret the 2m physical distancing rule.

Some of the data from this camera was incorporated into a very nice ROLSI blog post by Eric Laurier, Magnus Hamann and Liz Stokoe that I helped with about the emergence of the ‘social swerve’.

I thought others might find it useful to read a quick how-to about grabbing video from live cameras – it’s a great way to get a quick and dirty bit of data to test a working hunch or do some rough analysis.

There are thousands of live cameras that stream to youtube, but it can be a bit cumbersome to capture more than a few seconds via more straightforward screen capture methods.

NB: before doing this for research purposes, check that doing so is compliant with relevant regional/institutional ethical guidelines.

Step 1: download and configure youtube-dl

Youtube-dl is a command line utility, which means you run it from the terminal window of your operating system of choice – it works fine on any Unix, on Windows or on Mac Os.

Don’t be intimidated if you’ve never used a command line before, you won’t have to do much beyond some copying and pasting.

I can’t do an installation how-to, but there are plenty online:

Mac:

Windows:

I’ll assume that if you’re a Unix user, you know how to do this.

Step 2: copy and paste the video ID from the stream

Every Youtube video has a video ID that you can copy from the address bar of your browser. Here’s the one I used for the blog post mentioned above – which we’ve affectionately nicknamed the ‘kebab corpus’. The video ID is circled in red:

Step 3: use youtube-dl to begin gathering your video data

This bit is a little hacky – as in not really using the software as intended or documented, so I’ve created a short howto video. There might be better ways. If so, please let me know!

As I mention in that video – probably best not to leave youtube-dl running for too long on a stream as you might end up losing your video if something happens to interrupt the stream. I’ve captured up to half an hour at a time.

It’s possible to create scripts and automated actions for a variety of operating systems to do this all for you on a schedule – but if you need extensive video archives, I’d recommend contacting the owner of the stream to see if they can simply send you their high quality youtube archives.