arXiv preprint ·
Chart the Invisible: Interactive Perception for Hidden-Target Exploration in GUI Agents
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Despite rapid progress in command-line interface (CLI) tasks, recent LLM-based agents still struggle with reliable task execution through graphical user interfaces (GUIs). A key challenge is the limited content visible within a bounded viewport, especially on mobile devices, where many task-relevant targets remain hidden and can only be revealed through swipe interactions. Therefore, although most GUI agents support swipe as an action primitive, the lack of target visibility makes it difficult for them to decide when, where, and how to swipe to reveal the hidden content. Our quantitative study further shows that GUI agents incorrectly choose a click action in 47.8% of cases that require swipe exploration. To bridge this gap, this paper proposes equipping GUI agents with scrollable-region perception to support hidden-area exploration before action selection. To this end, we trained the first multimodal model for scrollable-region perception on a manually collected dataset of 667 UI screens, called Hagasu. By correctly identifying 78.5% of scrollable regions, experiments across nine GUI agents show that integrating Hagasu improves the mean end-to-end completion rate from 64.8% to 82.7% on explore-dependent tasks. To facilitate follow-up research, we release the swipe dataset and the source code of Hagasu.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents
ProCerDCL: Query-Efficient Black-Box Probabilistic Verification of Deep Learning Compiler Optimization
The Physical Cost of Context: Quality-Constrained Resource Accounting for Coding Agents
From Ripple to Boundary: Deterministic Impact Boundaries for LLM-Based Software Evolution
Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.