StereoPatch: Patch-Aligned RGB-Depth Fusion for Spatial Perception in Robot Manipulation

Abstract

Recent advances in robot imitation learning have produced visuomotor policies that predict actions directly from visual observations. Yet visually similar scenes can require different actions as target position, object height, or contact geometry changes. Pretrained RGB features may map these geometrically distinct states to similar policy inputs, while simply adding depth requires the policy to learn RGB-depth correspondence from the same limited demonstrations used to learn control. We introduce StereoPatch, a patch-aligned RGB-depth representation that binds registered metric geometry directly to the RGB patches used for action prediction. On a shared 2-D patch grid, asymmetric cross-attention incorporates depth information into the corresponding RGB features before action decoding. The resulting StereoPatch Tokens provide a geometry-aware visual representation that can condition general visuomotor policies without changing their underlying learning objectives. Across six real-robot tasks, StereoPatch achieves higher closed-loop success than appearance-only, geometry-only, raw RGB-D, and late-fusion baselines. Additional experiments across three simulation suites evaluate compatibility across visuomotor policy architectures, spatial generalization, and operating limits. Results suggest that resolving control-relevant geometric ambiguity benefits from aligning depth directly with the visual features used for action prediction, rather than supplying it as an independent modality. Project page: https://aus.bot/research/stereopatch/.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 14 September 2026

  Primitive-Informed Sampling-Based MPC for Multi-Fingered Dexterous Manipulation

  Emek Barış Küçüktabak, Karankumar Patel, Jinda Cui and 3 others
• arXiv preprint · 15 September 2026

  Dense to MoE Adaptation for Compact Vision Language Action Policies

  Muchun Niu, Shuang Chen, Yuzhou Wu and 1 other
• arXiv preprint · 14 September 2026

  Real-World Reinforcement Learning with MPC Scaffolding for Dexterous Manipulation

  Emek Barış Küçüktabak, Karankumar Patel, Zhaodong Yang and 3 others
• arXiv preprint · 15 September 2026

  UniDex-ViTac: Learning Unified Visuo-Tactile Dexterous Manipulation Policy from Human Video Data

  Hyesung Lee, Si-Hwan Heo and Sungwook Yang
• arXiv preprint · 14 September 2026

  HydroMap: Probabilistic Water Surface Elevation Mapping for Semantic Scene Representation in Inland Waterways

  Zhongbi Luo, Yunjia Wang, Herman Bruyninckx and 1 other

All new papers in Robotics