arXiv preprint ·
How AI Changes DevOps Performance: A Mechanism-Based Simulation
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
AI tools for software development have progressed from code completion to autonomous agents, yet evidence on their effects on DORA performance is mixed, mostly cross-sectional, and often conflates AI-generated code (AGC) with agentic AI (AGT). Objective. We examine how AGC and AGT, separately and jointly, affect deployment frequency, lead time, change failure rate, recovery time, and deployment rework rate. Method. We audit the evidence, formalise a two-construct model with four capability moderators, and implement a discrete-event simulation of a delivery pipeline. Six experiments include a factorial design, Shapley decompositions, sensitivity analysis over 600 parameter draws, a fixed-demand robustness test, and a staggered-adoption panel of 5,000 simulated teams with counterfactuals. Results. AGC increased failure rate, rework, and recovery time in both capability profiles. When AI increased change volume, AGC shortened lead time only where review capacity was spare (-13%) and lengthened it once review saturated; in the low-capability team, its deployment-frequency gain came entirely from unplanned rework. AGT improved deployment frequency (+31% to +34%), lead time (-8% to -22%), and recovery time (-29% to -38%), but its stability effect depended on autonomous CI repair masking defects, and escaped defects increased in the high-capability team. Capability reduced AGC's absolute failure-rate penalty (7.9 vs. 1.5 points) but not its relative penalty (+34% vs. +84%). Two-way fixed effects underestimated adoption effects by 8-16%; detecting the modelled stability effect required about 50 teams, and its capability moderation about 400. Conclusions. Within the model, DORA metrics respond to AI through queueing, batching, and oversight, not only code quality. We derive measurement rules, guardrails for agentic remediation, and a sample-size-informed field-validation protocol.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing
KernelZero: Co-Evolving Proposer and Coder for Continuously Improved GPU Kernel Generation
IChart2Code: Benchmarking Multimodal Large Language Models for Interactive Chart Code Generation
Compositional Safety Failures in Harness Evolution: Identification and Runtime Monitoring
On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.