arXiv preprint ·
Finite-Sample Convergence in Networked Average Reward MARL: Decentralization Pitfalls and Entropy Remedies
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We study the problem of average reward Multi-Agent Reinforcement Learning (MARL) where agents interact within a network. Each agent in the network conducts local updates with information within its $k$-hop neighborhood and collaboratively maximizes the overall reward of the entire network. We first provide impossibility results in achieving convergence to the global optimal joint policies in our setting. These impossibility results highlight the challenges of decentralized learning with local information in multi-agent systems, namely the issues of multiagency, where each agent independently optimizes its policy, and partial observability, where agents have limited visibility into the global state. Given these challenges which indicate that decentralized policy optimization can generally be certified only up to stationarity, we provide finite-sample convergence guarantees for a decentralized actor-critic algorithm with linear function approximation that converges to an approximate stationary point with small gradient norm. To overcome such impossibility results, we further show that adding fixed-state entropy regularization reshapes the objective landscape which leads to even stronger global convergence guarantees for our proposed actor-critic algorithm.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Scientific Liquidity Agents: An Agent-Based Model of Knowledge Arbitrage in Scientific Communities
Interpretable Early Termination of Web Navigation Agents via Closed Sequential Pattern Mining
Organizational Learning in Materials Discovery Campaigns: Which Databases, Calculations, and Agent Teams Pay Off
Drift No More: Spectral Locking for Long-Horizon Open-Weight Multi-Agent AI
Coati: Host-Grounded Execution for Installable Mobile GUI Agents
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.