arXiv preprint ·
Do Embodied VLMs Actually Use Their Scene-Graph Memory?
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Embodied agents that answer questions about their route need a memory of what they saw, and scene graphs are a natural form for it. A vision-language model (VLM) can receive such a graph as text or through a graph encoder, but accuracy alone cannot show whether the graph holds the episode's facts, reaches the model, or is actually relied on. In this diagnostic study, we introduce a graph-swap test for the last of these: each episode is given another episode's graph, and we count how many answers change. On an embodied question-answering benchmark with routes through two to seven rooms, we use it to compare ways of delivering scene-graph memory to a fine-tuned VLM. As text, the memory is used strongly, adding 30 points on route questions. As a graph, it is not, and we trace why: the encoder preserves the information, but standard pooled and late fusion discard it. Giving each node its own input token recovers part of it, a gain of 8.3 points on route questions that the swap test confirms comes from the graph. The test also exposes a shortcut that accuracy hides: when the same facts appear in the text, the model ignores its graph. The test thus offers a cheap check of whether a VLM uses its scene-graph memory, and our analysis points to how that memory should be delivered.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Robotics
Same Path, Less Time: Training-Free Fast Execution of Vision-Language-Action Models under Inference Latency
Late Autumn: LLM Agents Specialize a General Multi-Robot Planner for One Warehouse
Serving Agentic Robot Fleets: Workload Characterization, KV-Cache Thrashing, and Admission Control
Learning Purposeful Co-designs of Bio-inspired Quadrupedal Robots
Adaptive Surface Cleaning under Uncertain Cleaning Response
Get the few papers worth your time in Robotics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.