DexCast: Learning Real-World Dexterous Control from Human Videos

Abstract

Recent advances in 3D hand-object interaction (HOI) perception have opened new opportunities for learning dexterous robotic skills from human videos. However, existing dexterous imitation learning methods often rely on costly and manually teleoperated robot demonstrations, requiring substantial human effort. Therefore, we build DEXCAST, a framework that learns dexterous robotic hand control directly from monocular RGB-D videos by leveraging joint HOI reconstructions, without requiring any teleoperation. Furthermore, how to model high-dimensional robotic hand motion and extract temporal context to inform future action generation remains a key challenge. To this end, we propose a DexHand Motion Perception Module (DMPM) to capture inter-joint coordination and temporal dependencies in compact latent representations that condition our diffusion policy. Extensive real-world experiments demonstrate that DEXCAST achieves exceptional performance across diverse dexterous manipulation tasks.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 10 October 2026

  Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation

  Shihan Zhao, Henry Chen, Ya-Chuan Hsu and 7 others
• arXiv preprint · 11 October 2026

  Reinforcement Learning Based Heterogeneous Task Allocation Under Spatio-Temporal Environmental Uncertainty for Multi-Robot Teams

  Xi Lin, Yaxuan Li, Paul Szenher and 1 other
• arXiv preprint · 10 October 2026

  Shared-Control Teleoperation of Omnidirectional Aerial Robots for Continuous Valve Rotation

  Jinjie Li, Yicheng Chen, Haokun Liu and 1 other
• arXiv preprint · 11 October 2026

  LeHami: Symplectic Latent Dynamics for World Models

  Romina Yalovetzky, Randall Balestriero and Yann LeCun
• arXiv preprint · 10 October 2026

  From Field to Planning: Learning Temporal Polar Gradient Fields for Vision-based AAV Trajectory Planning in Dynamic Environments

  Haowen Gao, Zhan Chen, Yinuo Xu and 3 others

All new papers in Robotics