CFG-Policy: Unified Diffusion Policy Learning Framework via Classifier-free Guidance

Abstract

Diffusion policies are increasingly used in robotic control and manipulation, vision-language-action models, and world-model-based agents. Their ability to represent multimodal action distributions offers rich opportunities for exploration in reinforcement learning (RL). However, optimizing diffusion policies with RL can reduce the behavioral diversity and further limit their exploration capability. In RL post-training, directly fine-tuning a pretrained policy may also disrupt its acquired capabilities, while action-space residuals may lack the expressive power for multimodal exploration. Together, these issues reveal a common challenge across online and offline-to-online diffusion RL: improving task performance while retaining the behavioral diversity and capabilities needed for continued adaptation. We introduce CFG-Policy, a unified, plug-and-play framework that separates exploration and exploitation into two generative policies and combines their score or velocity fields during sampling through classifier-free guidance. For online RL, a frozen exploration branch sustains behavioral diversity while an exploitation branch learns from task rewards; for offline-to-online RL, a pretrained exploitation branch remains frozen while a trainable branch steers the composed behavior. CFG-Policy preserves the underlying RL objectives and improves performance with QSM and QVPO on 5 MuJoCo tasks and with Flow-SDE+PPO on 4 RoboTwin manipulation tasks. Project page is at https://zhy-liu.github.io/cfg-policy-page.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 10 October 2026

  Red-Teaming of Vision-Language-Action Models via Quality Diversity Environment Generation

  Shihan Zhao, Henry Chen, Ya-Chuan Hsu and 7 others
• arXiv preprint · 11 October 2026

  Reinforcement Learning Based Heterogeneous Task Allocation Under Spatio-Temporal Environmental Uncertainty for Multi-Robot Teams

  Xi Lin, Yaxuan Li, Paul Szenher and 1 other
• arXiv preprint · 10 October 2026

  Shared-Control Teleoperation of Omnidirectional Aerial Robots for Continuous Valve Rotation

  Jinjie Li, Yicheng Chen, Haokun Liu and 1 other
• arXiv preprint · 11 October 2026

  LeHami: Symplectic Latent Dynamics for World Models

  Romina Yalovetzky, Randall Balestriero and Yann LeCun
• arXiv preprint · 10 October 2026

  From Field to Planning: Learning Temporal Polar Gradient Fields for Vision-based AAV Trajectory Planning in Dynamic Environments

  Haowen Gao, Zhan Chen, Yinuo Xu and 3 others

All new papers in Robotics