Optimal Control with Learned Critics under Unmodeled State Dependencies

Abstract

Model Predictive Control (MPC) provides a structured and constraint-aware mechanism for decision-making, but its reliance on optimization-friendly analytical dynamics models limits its use in tasks with contacts and other hard-to-model state dependencies. Model-free reinforcement learning avoids explicit modeling assumptions but typically requires large amounts of interaction data. We present a learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia. The method augments a nominal analytical model with a residual dynamics network that learns missing state-dependent effects from data and combines the resulting planner with a learned action-value critic that injects long-horizon MDP structure into the local iLQR optimization. To make this practical at reinforcement-learning scale, we develop a GPU-accelerated batched iLQR solver that evaluates learned dynamics and critic networks inside the optimal-control loop and solves thousands of trajectory-optimization problems in parallel. The complete system is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynamics. Experiments on biased and incompletely modeled control tasks show that the approach improves closed-loop control performance while preserving the model-based structure needed for efficient constrained trajectory optimization.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 4 October 2026

  Tackling Sim-to-Real Mismatch Through Sampling-Based Disturbance Observers: From Analytical Models to Learned World Models

  Tianqi Zhu, Jun Yang, Jianliang Mao and 2 others
• arXiv preprint · 5 October 2026

  FOCUS: Fine-Grained Open-Vocabulary Change Detection for Uncertainty-Aware Semi-Static Scenes

  Can Xu, Mingfeng Yuan, Mahan Mohammadi and 1 other
• arXiv preprint · 4 October 2026

  GJK-CBF: Control Barrier Functions for Convex Rigid Body Collision Avoidance on SE(3)

  Yi-Hsuan Chen, Shuo Liu, Wei Xiao and 2 others
• arXiv preprint · 5 October 2026

  Spacecraft Rendezvous Trajectory Generation with Modular Constraints via Diffusion Model Composition

  Mariko A. Storey-Matsutani and Richard Linares
• arXiv preprint · 4 October 2026

  $R^2$-WAM: Repair-and-Reject Post-Training for World Action Models

  Ruiyan Xu, Haisheng Su, Sixu Lin and 4 others

All new papers in Robotics