arXiv preprint ·
Comparing Computer Use Behavior Across Multiple Levels of Abstraction
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Interaction traces record how people and agents use computers, and comparing them can reveal usability barriers, differences in expertise, and human--agent similarity. Yet traces are recorded as low-level, interface-specific actions, making comparisons difficult across tasks, interfaces, and input modalities. Existing approaches either compare these actions directly, reduce traces to aggregate outcomes, or summarize them at higher-level goals, obscuring where behavior differs. We present MAP (Multi-level Action Parsing), a hierarchical semantic representation and annotation pipeline that maps traces into a common vocabulary across linked levels of abstraction. MAP compares traces by sequence, recurring patterns, and effort, localizing differences by type and abstraction level. A technical evaluation across three computer use datasets shows high agreement with manual annotations and broad coverage of behavior. Applying MAP shows that sighted and blind users increasingly converge at higher levels despite navigation differences, while agents omit corrective, help-seeking, and recovery behaviors present in human traces.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Human-Computer Interaction
When Better Opportunities Backfire: Service Worker Responses to Parallel AI Deployment in Customer Acquisition
Relational Complexity: Making Sense of Social Companion AI Use
When Care Expands and Control Declines: Understanding Digital Privacy in End-of-Life Care
Bias and Reliability in Learning-Style Inference from Student--AI Dialogues: An Evaluation of GPT-4o and Gemini 2.0 Flash
Cling: Supporting Cross-Page Coordination with Human-Directed, In-Situ, and Small-Step Automation
Get the few papers worth your time in Human-Computer Interaction every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.