arXiv preprint ·
From Noisy Retrieval to Decision Evidence: Silver Evidence Demonstrations for In-Context Learning
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
RAG and retrieval-augmented few-shot ICL can struggle when decision-relevant evidence is buried in long, noisy context. Retrieved passages and demonstrations may contain redundant or distracting information. Although expert-labeled development examples may identify the text supporting each decision, these annotations are typically used only as answer-side rationales rather than as supervision for filtering noisy retrieval. We propose silver-evidence demonstrations, which replace raw retrieved passages in ICL examples with generated, target-conditioned evidence summaries. For each labeled development example, an LLM generates silver evidence without access to the expert label or gold evidence; this evidence is then paired with the expert label and gold evidence where available. At test time, a single structured call generates silver evidence from retrieved passages and then predicts the final label and decision evidence. Across EI 2.0, SciFact, synthetic policy retrieval, and HICRIC/IMR data, our full method improves over equally sized gold-only ICL (label-only for policy) on three of four datasets and improves evidence F1 on all annotated datasets. It also achieves higher mean label accuracy than InstructRAG-style ICL on all four datasets. These results suggest that replacing demonstration-side retrieved passages with compact, target-conditioned silver evidence is particularly useful for decisions based on long and noisy text, where it reduces distracting context while preserving decision-relevant signals and expert supervision.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments
Examining form-based linguistic productivity in people and large language models
X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints
Perturbation-Based Data Selection for Medical Question Answering
How Much Do You Trust A Dog? Quantifying Hint-Following in Reasoning Language Models
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.