arXiv preprint ·
Sandman: A Sandbox Service for Production Agentic Reinforcement Learning
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Agentic reinforcement learning (RL) needs stateful environments for tool execution and evaluation. Training jobs request thousands of environments in bursts that can overwhelm shared cluster services, while cancelled or timed-out requests can leave resources running. We present Sandman, a sandbox service built and operated at Amazon for large-scale agentic RL workloads in production on shared Kubernetes. A small daemon mounted into unmodified task images preserves files and processes across calls. Sandman paces creation requests, checks sandbox readiness, and retains cleanup responsibility after callers time out or cancel. Production deployments created 3.1 million sandboxes for 122 coding-agent training runs in two weeks. Peak production concurrency was 11,858 sandboxes. In one 10,000-request multi-client burst, Sandman made half the sandboxes usable in 116 seconds and all in 308 seconds. Full delivery took 44% less time than a tuned configuration of agent-sandbox, an open-source sandbox manager for Kubernetes. Sandman left no sandboxes at client exit. On software-engineering task images, median no-op command latency was 2.0 ms, compared with 14.3 ms for agent-sandbox and 203.5 ms for OpenSandbox. We also measure provisioning costs and sandbox delays within training runs, and report design trade-offs and operational lessons from production use and comparisons with three open-source systems. We will open-source Sandman's daemon, provider and pool.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Distributed & Parallel Computing
DICE: Decentralized Incentive-driven Federated Learning for Efficient & Scalable Customization of Pretrained Models in Web3
ZenKO: Guided and Iterative Quantum-Classical Method for Molecular Electronic Structure using Sample-Based Quantum Diagonalization
More Is Better? Exploring Latent Correspondence in Federated Hypergraph Learning
ARK: Avoiding Routing Collisions for KV Cache Transfer in Disaggregated LLM Inference
Learning to Formulate: Agentic Hardware-Software Co-Design of AI Infrastructure through Progressive Optimization and Verification
Get the few papers worth your time in Distributed & Parallel Computing every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.