arXiv preprint ·
RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1145/3767308.3836574
- Original
- Read it at arXiv
- Topics
Abstract
Graphical User Interface (GUI) grounding is a fundamental perception task for multimodal agents, enabling them to interpret natural language instructions and interact with digital interfaces. Existing methods face a fundamental trade-off between accuracy and efficiency: direct full-image inference often fails to capture small or visually similar UI elements, while multi-crop strategies improve localization at the cost of multiple expensive Vision-Language Model (VLM) calls per query. To address this challenge, we propose RankGround, a two-stage framework that achieves accurate GUI grounding with a single VLM call per query. Central to our approach is GroundRanker, a lightweight multimodal reranker that identifies the most promising crop from a dense candidate set. Because no off-the-shelf ranking dataset is available, we construct ranking supervision data from existing grounding datasets. A strict containment criterion and boundary-aware positive augmentation improve alignment and spatial coverage in cluttered layouts. GroundRanker is then trained with a two-stage curriculum: a pointwise objective first learns coarse containment, and a listwise objective refines subtle semantic and spatial distinctions among visually similar crops. Experimental results show that RankGround consistently outperforms strong baselines while reducing computational cost. It achieves 1.4 times faster inference and improves localization accuracy by 5.5% on average over the second-best method across all backbones and screen scales, establishing a new state of the art in both efficiency and precision for GUI grounding.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
A11yLTLNav: Automatic Detection of Accessibility Navigation Failures
Value Faces: Surfacing How Self-Presentation Shifts Across Relationships
Measuring AI Leadership: Development and Validation of a Multidimensional Measure for AI-Native Organizations
Full-Duplex Speech Models Take the Floor When Asked, Not When Needed
Gaze as Evidence for Common Grounding: A Cross-Corpus Analysis of MapTask and MUNDEX
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.