arXiv preprint ·
Beyond the Prompt: Linking What Developers Ask, Do, and Understand with Coding Agents
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Coding agents can now change code for developers, who describe goals, supply context, and respond to the agent's work. Yet prompts, screen activity, and task success each tell only part of this story. We present Say, Do, Understand, an end-to-end workflow for analyzing what developers write to an agent, what they do while it works, and what they can explain afterwards. The workflow has five stages (Capture, Prepare, Analyze, Integrate, and Interpret) and three instruments: a prompt codebook, a scheme for coding screen-recorded activities and events, and separate rubrics for explaining the process and the solution. We applied it in an observational study of ten experienced developers who used GitHub Copilot on an unfamiliar codebase. Crossing task performance with understanding produced four personas. The two measures agreed for eight developers but split for two: one passed most tests but could not explain the solution, and another passed few tests but explained it well. In this sample, the personas that most often asked the agent to check its work spent the least time testing on their own. These patterns are descriptive and do not generalize beyond the sample. We recommend the workflow to computing educators, industry practitioners, and researchers to adapt and evaluate human--AI communication in software engineering.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Software Engineering
KernelZero: Co-Evolving Proposer and Coder for Continuously Improved GPU Kernel Generation
Towards Certificate-Driven Software Porting: A Self-Improving Agentic Harness for Scientific Program Optimization
Compositional Safety Failures in Harness Evolution: Identification and Runtime Monitoring
Faultless: A Program Equivalence Technique for Validating and Evaluating Neural Decompilers
On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation
Get the few papers worth your time in Software Engineering every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.