arXiv preprint ·
A Multi-Modal Robotic Framework for Physical GUI Interaction Testing
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.1145/3774748.3787623
- Original
- Read it at arXiv
- Topics
Abstract
Automated testing of mobile applications remains challenging when realistic user interactions must be reproduced and GUI behavior verified without relying on invasive mechanisms such as instrumentation or ADB-level control. Existing approaches, including DLD and R-DLD, detect data loss triggered by activity restarts but still depend on system-level access, which limits their applicability on real devices. This doctoral research proposes RFNIT, a multi-modal, fully non-invasive robotic framework for physical GUI interaction testing on smartphones. RFNIT supports end-to-end automation by perceiving the device state through external sensing and executing physical interactions, enabling human-like actions while preserving reproducibility and execution accuracy. The framework integrates a vision-based perception module, an LLM-driven reasoning agent, and a robotic actuator capable of performing touch, swipe, and typing gestures. Multimodal data, including external camera images and OCR-extracted text, are combined to identify GUI elements and guide subsequent testing decisions. Preliminary results indicate that RFNIT accurately recognizes buttons, text fields, and virtual keyboards from external camera input, achieving over 85\% visual detection accuracy. Ongoing and future work will focus on improving robustness, repeatability, and coverage of GUI states during long-running testing sessions.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
AIFaultBench: A Reproducible Benchmark of Real-World Faults in AI Software Systems
DERED: Enhancing Third-Party Library Detection in Binaries via Deceptive Reuse Mitigation
RepGen: Reproducing Deep Learning Bugs through Context-Aware Code Generation and Verification
AgentPRO: Coordinating Compiler Optimizations for LLM Agent Workflows
One Verdict Is Not Enough: Regression Testing Learned World Models and Their Test Harnesses
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.