DRHeC: Differentiable Rendering for Hand-Eye Calibration with RGB-Based Gradients

Abstract

Accurate hand-eye calibration is crucial for precision manipulation. Traditional methods rely on markers, with their precision dependent on marker accuracy and observability. In contrast, markerless methods, such as learning-based approaches, use deep neural networks to directly extract keypoints or features from images, enabling the computation of hand-eye transformation with a single image and without the need for physical markers. Recently, differentiable rendering-based methods for hand-eye calibration have leveraged physical models to render binary masks and compare them with observations, enabling hand-eye calibration without fiducial markers in the calibration stage and providing interpretable optimization. While the state-of-the-art differentiable rendering methods achieve remarkable accuracy, the use of binary masks can result in the loss of internal profile details, reducing precision. Additionally, these methods can also suffer from unstable optimization and local minima. In this study, we propose a novel RGB-based differentiable rendering framework that provides richer geometric and appearance cues by incorporating color and mask geometric features, thereby improving calibration accuracy and optimization stability. Additionally, we propose a mask-guided image-to-image translation method to ensure explicit preservation of color and geometric consistency throughout the translation. Our approach is validated through both simulation and real-world experiments, with results demonstrating strong accuracy and robustness and clear improvements over existing differentiable rendering methods. Our method achieves a grasping success rate of 88.9% and insertion success rate of 57.4% on the UR5e real-world experiment, outperforming the state-of-the-art differentiable rendering hand-eye calibration method EasyHeC by 46.3 and 48.1 percentage points, respectively.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 29 September 2026

  Text2Sim: Agentic Physics-Based Simulation Generation with Distilled Expertise

  Xiaoyu Xiong, Tsun-Hsuan Wang, Yi-Ling Qiao and 2 others
• arXiv preprint · 30 September 2026

  Lens Flare Removal and Reconstruction

  Tarun Yenamandra, Jonathon Luiten, Daniel Cremers and 1 other
• arXiv preprint · 30 September 2026

  Diffusion Editing with Soft Mask: Pixel Level Redo of Image and Video with Adjustable Strength

  Candi Zheng and Yuan Lan
• arXiv preprint · 29 September 2026

  NRF-GS: Neural Residual Fields for Expressive and Compact Gaussian Splatting

  Pratik Singh Bisht and Andreas Kolb
• arXiv preprint · 30 September 2026

  Faithful Chart Generation for Multimodal Deep Research: Frame-Evidence Co-Adaptation

  Yuxin Yue, Yingchen Zhang, Ruqing Zhang and 3 others

All new papers in Robotics