arXiv preprint ·
Using Evidence-Centered Design to Guide Prompt Engineering in Measurement: An Illustration with Automated Essay Scoring
- Published
- Source
- arXiv preprint, posted before peer review
- Published in
- arXiv
- DOI
- 10.48550/arxiv.2610.15231
- Original
- Read it at arXiv
- Cited by
- 0 works, as OpenAlex counts them
- Topics
Abstract
Generative artificial intelligence (GenAI) models, particularly large language models (LLMs), are increasingly used in educational settings for tasks such as item generation, automated scoring, and feedback delivery. The quality of these outputs depends not only on the AI model but also on how users construct the prompts that guide model behaviors. Current prompt engineering practices are largely ad hoc, relying on trial and error rather than principled design. This lack of structure introduces validity threats including construct misrepresentation, construct-irrelevant variance, and limited reproducibility. We propose that Evidence-Centered Design (ECD), a framework in educational measurement for structuring valid inferences from observed performance, can serve as a principled foundation for prompt engineering in educational applications. We map the three core ECD models, the student model, the evidence model, and the task model, onto the components of educational prompt design, and describe a workflow for ECD-based prompt engineering. We derive five design principles from this mapping and present an empirical study contrasting ECD-designed prompts with state-of-the-art (SOTA) prompting baselines on an automated scoring task. Our empirical study demonstrates the usefulness of this framework on automated scoring. Results suggest that treating prompting as assessment design rather than informal instructions strengthens the validity, reliability, and fairness of AI-generated educational outputs.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
Impact of Expertise and Trust on AI-assisted Work
ArguScale: Evidence-Grounded Human-AI Sensemaking Across Concurrent Live Conversations
sGLP: Simulating People with Stochastic Agents
AuthScribe: Supporting Agency and Authorship in LLM-Assisted Reflective Journaling
Rethinking Opinion Measurement Interfaces for Human-in-the-Loop Optimization
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.