OATH: Object-Agnostic Tracking for Dexterous Hands, Enabling Rapid Physics-based Retargeting

Abstract

Retargeting human demonstrations into dynamically feasible dexterous hand trajectories requires resolving complex contact physics, yet current methods either omit physical constraints or rely on computationally expensive sampling-based optimization in simulation. Reinforcement learning has previously been used to achieve planning and control through contact dynamics; however, standard policies require object-specific training and fail to generalize across arbitrary geometries. This work reframes physics-based retargeting as a residual learning problem, where we demonstrate that zero-shot generalization across objects relies on structured state representations. Specifically, formulating the observation space to include a localized contact-error signal guiding each robot link to its target coordinate on the object surface enables a single, unified residual policy to successfully grasp novel, unseen object geometries. Rolling out this policy in simulation on a kinematic reference trajectory yields a physics-based retargeter, OATH. Operating at 0.4 seconds of compute per second of reference, OATH achieves 2.4x faster-than-real-time performance and a 60-350x reduction in computational cost compared to sampling-based methods. The system recovers 58% of held-out and 54% of unseen-object DexYCB trajectories while maintaining a sub-2 cm tracking error.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 9 October 2026

  Cubic Spline Interpolation for Smooth Trajectory Generation in Autonomous Motion Systems

  Aarush Mandayam, Sriram Aditya Gundla and Elliott Tammaro
• arXiv preprint · 10 October 2026

  Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation

  Shihan Zhao, Henry Chen, Ya-Chuan Hsu and 7 others
• arXiv preprint · 9 October 2026

  VLMs are Native Action Models: Unleashing the Manipulation Capabilities of VLMs with Explicit Action-Space Grounding

  Jingkun Yi, Shujie Zhang, Zirui Zhou and 8 others
• arXiv preprint · 10 October 2026

  Shared-Control Teleoperation of Omnidirectional Aerial Robots for Continuous Valve Rotation

  Jinjie Li, Yicheng Chen, Haokun Liu and 1 other
• arXiv preprint · 9 October 2026

  Reinforcement Learning via Fixed-point Flow Matching

  Denis Blessing, Onur Celik and Gerhard Neumann

All new papers in Robotics