arXiv preprint ·
Evidence Sufficiency and Model Comparisons in MexRegQA: An Item-Level Audit of Mexican-Spanish Regulatory QA
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Similarity-gated coverage does not establish that retrieved evidence contains the facts needed to answer a question. We audit Mexican-Spanish financial-regulatory QA and find that 35 of 135 covered items lack some or all load-bearing facts. On the post-hoc full-evidence slice (n=98), eleven prompt-matched models score 95-98/98 and none differs significantly from the anchor; this does not establish equivalence. Removing evidence-defective items changes the nominal significance status of 14 of 55 pairwise comparisons. On 56 zero-evidence items, accuracy spans 33.9%-89.3%. Removing evidence from the audited items under an abstention-permitting prompt reduces accuracy by 63-81 percentage points in all ten tested legs, with no recovered items. Building on sufficient-context research, we contribute MexRegQA v1.0: 192 items with provenance and evidence-status labels, frozen retrieval evidence for 191 evaluated items, predictions, judge outputs and offline checks. The audit was outcome-aware, the later model/panel sweep reused the same questions, and all reported scores and labels rely on machine evaluation without independent human validation. We propose releasing evidence-status labels alongside grounded QA benchmarks so that model comparisons can be examined separately from evidence availability.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Machine Learning
Cognitive Maps in Large Language Models
Learning the Geometry of Materials Space with Trained Neural Tangent Kernels
Text Dominance in Paralinguistic Conflict: Tracing Suppression Through Audio LLM Circuits
The Router Already Knows
Adversarial Post-Training of Foundation Models for High-Frequency Trading
Get the few papers worth your time in Machine Learning every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.