Can Jev Improve Demonstration Retrieval for In-Context Learning?

Abstract

Demonstration selection plays a crucial role in the effectiveness of in-context learning (ICL) for large language models (LLMs). Extensive research has explored various retrieval strategies to identify informative demonstrations and improve downstream performance. Recently, the emergence of Jev, a decision-oriented language model capable of producing probabilistic judgments, offers a new perspective on evaluating demonstration utility. This motivates us to investigate a fundamental question: Can Jev improve demonstration retrieval for in-context learning? In this paper, we demonstrate that Jev can effectively enhance demonstration retrieval by directly assessing the utility of candidate demonstrations for a given query. However, applying Jev to large-scale retrieval introduces substantial computational overhead, as each candidate requires an additional decision-making inference. To address this limitation, we propose a simple yet effective knowledge distillation approach that transfers Jev's utility estimation capability to a lightweight multilayer perceptron (MLP). By learning to approximate Jev's probabilistic judgments, the distilled model enables efficient demonstration ranking while maintaining competitive retrieval performance. Our study highlights the potential of decision-oriented language models for demonstration selection and provides an efficient framework for integrating their decision-making capabilities into ICL.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Artificial Intelligence

• arXiv preprint · 31 October 2026

  Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation

  Gustavo Tironi, Felipe Maia Polo and Diego Mesquita
• arXiv preprint · 1 November 2026

  A Self-Exploring and Self-Evolving Paradigm for Adaptive Edge AI Development

  Leming Shen, Qiang Yang, Xinyu Huang and 3 others
• arXiv preprint · 31 October 2026

  Threat Modeling Intimate Partner Abuse in Agentic Systems

  Natalie Grace Brigham, Eugene Bagdasarian, Franziska Roesner and 1 other
• arXiv preprint · 1 November 2026

  Toward Executable Multi-framework Front-end Code Generation with Self-Correction

  Linxiao Li, Haoran Ma, Chenyue Wang and 3 others
• arXiv preprint · 31 October 2026

  Spotting the Artificial: Dialogic Humor Improvisation in Humans and LLMs

  Lizeth Palacios Patiño, Ximena Gutierrez-Vasques and Daniel Rodriguez

All new papers in Artificial Intelligence