Conditional Flow Models for Offline Maximum Entropy Reinforcement Learning

Abstract

Offline maximum entropy reinforcement learning seeks to estimate the optimal soft action-value function \(Q^*\) from a fixed dataset without further interaction with the environment. Although \(Q^*\) is characterized by the soft optimal Bellman equation, the corresponding Bellman operator cannot be evaluated directly because both the reward distribution and the transition kernel are unknown. Consequently, estimating \(Q^*\) requires the joint treatment of environment modeling and value function learning. We propose a framework that addresses these tasks separately. Conditional diffusion models based on probability flow ODEs estimate the conditional reward distribution and transition kernel. The resulting distributional estimators define a data driven approximation of the soft optimal Bellman operator, whose empirical Bellman residual is minimized over a class of deep neural networks to estimate \(Q^*\). We establish nonasymptotic guarantees for the proposed method. For operator learning, we derive total variation error bounds for the conditional flow estimators that account explicitly for score estimation, numerical discretization, and terminal smoothing. For soft value learning, we first use the soft Bellman equation to show that $Q^*$ inherits Hölder regularity from the conditional reward and transition densities. For any Hölder index $β$, the value learning stage achieves an excess Bellman residual risk bound of order $\widetilde{\mathcal O}\big(n^{-\frac{2β}{d+2β}}\big),$ where $d$ is the total dimension of the state and action spaces. In contrast to many existing theoretical analyses of deep reinforcement learning, our results do not require Bellman completeness. Numerical experiments support the theoretical findings and demonstrate the effectiveness of the proposed method.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Machine Learning (Statistics)

• arXiv preprint · 23 October 2026

  Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data

  Joonhyeok Jeong, Jy-yong Sohn and Kibok Lee
• arXiv preprint · 24 October 2026

  Faithful Federated Tabular Data Synthesis

  Zilong Cao, Hai Zhang and Xuan Bi
• arXiv preprint · 23 October 2026

  DBCM: Dual-Branch Calibration Module for Test-Time Adaptation in Time Series Forecasting

  Jaebyeong Jeon, Kunwoo Kang, Sang Yup Lee and 1 other
• arXiv preprint · 24 October 2026

  Online Covariance Estimation for Averaged Stochastic Gradient Descent on Riemannian Manifolds

  Jingrong Guan, Liwei Jiang and Sen Na
• arXiv preprint · 24 October 2026

  Causal Inference in Metric Spaces via Distance Profiles

  Zheye Song and Zhiheng Zhang

All new papers in Machine Learning (Statistics)