arXiv preprint ·
Testing with Coding Agents: A Large-Scale Comparison of Agent-Supported and Human-Authored Pull Requests
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Coding agents increasingly contribute tests through pull requests that developers then maintain, so how testing work is divided affects long-term maintainability. Objective. We compare pull requests in which developers and coding agents collaborate (agent-supported) with pull requests without detectable agent involvement (human-authored) along five dimensions: the frequency and extent of test changes, their type and context, test structure, test smells, and code coverage. Method. We analyze 114,134 pull requests merged between January 2025 and April 2026 in 391 Java repositories, identifying five coding agents through pull-request-level and commit-level signals. Since agent adoption varies across projects, we compare the two groups within repositories with mixed-effects models that control for change size. Results. Pooled across repositories, both groups change test code almost equally often; within repositories, the odds that an agent-supported pull request does so are 1.65 times higher. Agent-supported pull requests add tests more often (odds ratios 1.50-1.58) and modify (0.70-0.76) and delete (0.51-0.77) them less often. We detect no difference in most structural characteristics of the tests they add, which more often have an oracle, nor, in the 95 repositories we could build, in the coverage of their production changes; in test-only pull requests, however, their tests cover less of the code they exercise. Differences in test smells are borderline and not reproduced for individual test methods. Conclusions. Agent participation is associated with a different mix of testing work, which pooled comparisons across projects can hide or reverse; its relationship with test quality remains uncertain.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
Thinking-in-Bets at Test Time Reduces Sycophantic Revisions in LLM Agents
ProCerDCL: Query-Efficient Black-Box Probabilistic Verification of Deep Learning Compiler Optimization
The Physical Cost of Context: Quality-Constrained Resource Accounting for Coding Agents
From Ripple to Boundary: Deterministic Impact Boundaries for LLM-Based Software Evolution
Arya Forge: Auditing Validation and Release Approval for Financial Instruction Data
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.