OpenAlex ·
GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space
- Published
- Source
- OpenAlex
- DOI
- 10.48550/arxiv.2608.30188
- Original
- Read it at the publisher
- Topics
Abstract
Large Language Models (LLMs) show great potential as clinical agents, yet existing benchmarks reduce clinical workflows to static predictions or unconstrained Markov Decision Processes (MDPs) with coarse action sets. To address this, we introduce GPAgentBench-2K, the first Constrained MDP (CMDP) LLM-agent benchmark for primary-care clinical decision-making, constructed from expert-validated records of real-world GP encounters. Our environment models a full spectrum of six foundational clinical actions, imposes a topological workflow prior over the action space, and operationalizes safety-informed abstention as a first-class outcome. Evaluating 16 state-of-the-art LLMs reveals a significant performance degradation as the action space scales. Crucially, we uncover a clinical quality-safety gap: even frontier models with the highest diagnosis accuracy violate safety constraints in over half of high-risk cases. Finally, we establish a reference point using Constrained Group Relative Policy Optimization (C-GRPO), and show that while explicitly modeling constraints improves performance over unconstrained RL methods, it remains far from clinically acceptable safety.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
Camera trap classification with deep learning under ground truth uncertainty
JENGA: Exploiting Counter-Based RowHammer Countermeasures to Break Real-Time Predictability
Learning Dynamics of Logits Debiasing for Long-Tailed Semi-Supervised Learning
Potential-Guided Particle Steering for Negation-Constrained Dexterous Grasping
Optimization Landscape Geometry in VQE for Frustrated Quantum Spin Models
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.