arXiv preprint ·
Measuring "Success" In the Transition from Pre- to Post-LLM Conversational Agents through the Lens of Interaction & Explanations for Older Adults
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1145/3838685.3849860
- Original
- Read it at arXiv
- Topics
Abstract
Conversational agents (CAs) for older adults have moved through at least two distinct eras, split by the emergence of large language models (LLMs) at the end of 2022. The first era was defined by commercial smart speakers such as the original Google Home and Amazon Alexa, built on rigid, rule-based dialogue. Interactions were single-turn, user-initiated and "successful" if and only if a command matched one of a finite set of pre-templated intents. The current era is marked by LLM-powered systems capable of open-ended, multi-turn conversation, with capacity for personalization and foundations for more proactive interaction. This shift has not only transformed what CAs can do for users, but also changed what researchers can measure when it comes to understanding "success" of user interactions with CAs. In this case study, we reflect on over 8 years of research spanning more than 100 participants across multiple field sites to trace how our own working definitions of "success" have evolved. We reflect on two completed bodies of work with pre-LLM CAs and an in-progress study with a post-LLM CA. We discuss how our definition and measurement of success have evolved during this transition and also ask open questions to the community about how to optimally measure "success" in the post-LLM CA era. This research is deeply applied and subject to the messiness of real-world deployments. Our intended contributions are deliberately practical, a combination of methodological considerations and reflection questions to support researchers engaging with this new wave of technologies.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Human-Computer Interaction
Cross-Scale Spatial Reasoning: A Learning-Centered Framework for Designing Immersive Learning Environments
VisCodeBench: a new benchmark dataset for Text-to-Vis reflecting real-world practice
Gamification Box: Automated Personalized Rewards for Improved Task Experience
IMUdrape: Generating Garment-Mounted Virtual IMU Data from Text through Physics-Based Cloth Simulation
Endless VR Walk: Dynamic Mapping of Limited Physical Spaces into Infinite Linear Paths
Get the few papers worth your time in Human-Computer Interaction every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.