arXiv preprint ·
It's Complicated: Factors Influencing the Effectiveness of LLM-Generated Feedback in a Live Programming Course
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
We report on three iterations of an LLM-generated feedback system deployed in a live web development course, spanning three underlying models and three feedback formats. We ask how feedback presence and its characteristics (length, complexity, format) relate to student performance, whether prior feedback exposure is associated with final-project performance, and how performance changes after first exposure. Feedback was associated with more favorable immediate outcomes in several comparisons, but the pattern varied by outcome and period. In pooled models, longer feedback was associated with lower next-submission correctness, more remaining attempts, and longer resubmission time. Prior same-part feedback was associated with lower final-project solve probability. Student-level comparisons found differences on final-project steps begun before exposure, while within-student changes across steps were uncertain. These comparisons do not identify the causal effect of feedback. Changes on standard exercises were more favorable among exposed students, but the groups' different course-stage splits limit causal interpretation. Bounded message length and student choice of format are directions for experimental evaluation. Our final-project analysis captures follow-up performance rather than directly measuring learning or understanding.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Human-Computer Interaction
When Better Opportunities Backfire: Service Worker Responses to Parallel AI Deployment in Customer Acquisition
UnexpectedParrots: Iterating Behavioral Experiment Design with In-Silico Simulations
When Care Expands and Control Declines: Understanding Digital Privacy in End-of-Life Care
Simulating Infant Senses with Age-Tuned Networks
Cling: Supporting Cross-Page Coordination with Human-Directed, In-Situ, and Small-Step Automation
Get the few papers worth your time in Human-Computer Interaction every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.