RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions

Abstract

Learning controllable world dynamics from heterogeneous interaction videos is challenging because robot embodiments use incompatible action spaces, while human videos lack native control labels. We present RynnWorld-Latent, a cross-embodiment video world model that uses the 3D-grounded latent actions introduced by RynnLAM as a shared conditioning interface. This representation enables joint pretraining on human interaction videos, robot demonstrations, and simulation rollouts without requiring a common native action space. Built on a pretrained video transformer, RynnWorld-Latent combines global action tokens with an identity-initialized local pathway that aligns actions with future video latents. We pretrain on 23 datasets comprising approximately 553.5M frames and adapt the model to target robots through post-training. The framework supports two data-generation routes: digital teleoperation uses target robot commands to generate action-video demonstrations, while latent-action synthesis transfers source motion to an edited initial frame and pairs the resulting rollout with retargeted pseudo robot actions. Experiments show improved video prediction metrics on public benchmarks and two real-robot platforms, alongside qualitative action following when different action sequences share the same initial observation. Synthetic mid-training improves policy success rates on LIBERO and LIBERO-Plus, and both generation routes improve real-world policy success rates over real-demonstration-only training in the evaluated settings. These results support universal latent actions as a scalable interface for learning controllable visual dynamics across heterogeneous embodiments.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 27 October 2026

  A Singularity Avoiding Track Model for Race Car Optimization

  Adwait Verulkar, Behzad Danaei, Srikanth Sivaramakrishnan and 1 other
• arXiv preprint · 28 October 2026

  Same Label, Different Interface: Beyond Action-Space Labels in Robot Learning

  Arshia Sangwan and Zhongyang Zhang
• arXiv preprint · 27 October 2026

  Ground Sample Distance Constrained k-Coverage Viewpoint Planning via Set Multi-Cover Formulation

  Waleed Bin Khalid and Chul Min Yeum
• arXiv preprint · 28 October 2026

  Minimum-Time Escape from the Conic Field of View of a Reeds--Shepp Pursuer: the Faster Evader

  Ubaldo Ruiz
• arXiv preprint · 27 October 2026

  Whole-Body Humanoid Control via Stochastic Motion Tokenization

  Michael Drolet, Oleg Arenz and Jan Peters

All new papers in Robotics