arXiv preprint ·
FIRE: Failure Isolation and Reliability for Production-Connected Experiments at Scale
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Pre-production changes are usually tested in isolated environments to prevent their side effects from reaching production. Some record-and-replay platforms allow a production-connected mode: candidate changes receive recorded production requests while selected downstream calls reach production dependencies. This improves fidelity before deployment, but it also introduces reliability risks. At hyperscale, experimental traffic can fan out across shared services, trigger latent defects, or send malformed input to systems serving user requests. In this paper, we show how production-connected experimentation can operate reliably in a large-scale recommendation serving system. To contain downstream effects without sacrificing realistic dependency behavior, the platform uses three complementary guardrails. Isolated tiers redirect replay traffic away from shared dependencies where replication is practical. Dependency monitors observe downstream health and experiment-attributed errors and terminate experiments after sustained violations. Traffic pinning limits experimental calls to a designated subset of downstream workers when full replication would require scarce capacity. Isolated ranking tiers process billions of replayed requests per day using under 1% of the corresponding serving capacity. Over 90 days, these tiers experienced more than 200,000 abnormal worker exits without directly affecting production-serving workers. During a separate three-month window, dependency monitors cancelled approximately 5% of experiments for the measured ranking service and accounted for 87% of its cancellations. Dependency-aware triggering also reduced code-review experiment executions by 25%. These results show that high-fidelity, production-connected experimentation can operate at scale when downstream effects are isolated where possible, bounded where necessary, and continuously monitored and attributed.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
GC-MoE on SM120: Overlapping Expert-Parallel Token Dispatch with Warp-Level FP8xFP4 Block-Scaled MMA on Consumer Blackwell
Scaling VLA Serving Under Robotic Execution Constraints
LoCoJAX: Host-Local Coordination for Small-Model RL on TPUs
Instruction-Aware Transfer of GPU Implementation Crossovers: a Tensor-Product Discontinuous Galerkin Operator on GB200, H100, and A100
Snoopie: A Multi-GPU Communication Profiler and Visualizer
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.