Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation

Abstract

Automatic fact-checking systems assess the veracity of claims given evidence from relevant documents. Large Language Models (LLMs) have demonstrated strong performance in fact-checking due to their general reasoning capabilities. However, it remains unclear whether they faithfully make use of the evidence provided to reach veracity judgments or rely on parametric knowledge. To investigate this, we introduce Fact-Ablated Evaluation (FAE), a new evaluation framework that iteratively ablates the cited evidence to assess whether LLMs revise their predictions accordingly. Our empirical results show that current off-the-shelf LLMs as fact-checking systems rely more on their parametric knowledge than on the evidence provided. To bridge this gap between prediction accuracy and evidence grounding, we propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification through counterfactual evidence supervision for the LLM-as-verifier models. Experiments on four fact-checking datasets across different domains demonstrate that models trained with REAL obtain superior evidence-dependent capabilities compared to standard fine-tuned models. Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency, while REAL encourages veracity predictions to remain more closely tied to the availability of supporting evidence.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Computation & Language / NLP

• arXiv preprint · 8 September 2026

  Artificial Intelligence-Assisted Digital Inventory of Cultural Heritage & Traditional Knowledge: Case for Indonesian Open Digital Library of Culture

  Hokky Situngkir
• arXiv preprint · 9 September 2026

  High-probability guarantees for linear accessibility in feature superposition

  Enrico Vompa
• arXiv preprint · 8 September 2026

  Snugi-AI-v2 @ eRisk 2026 Task 2: Early Depression Detection via a Learned Stopping Policy with Sustained Confidence Gate

  Yuwen Chiu
• arXiv preprint · 9 September 2026

  Should I Be Polite to My LLM Relevance Judge? Tone as a Severity Operating-Point Shift

  Tian Zhang and Meng Li
• arXiv preprint · 8 September 2026

  Cassette: Case-to-Case Structural Distillation for Efficient Legal Case Retrieval

  Yanran Tang, Ruihong Qiu, Hongzhi Yin and 2 others

All new papers in Computation & Language / NLP