arXiv preprint ·
SAUCE: Summary Analysis Using Conversation Entailment
- Published
- Source
- arXiv preprint, posted before peer review
- DOI
- 10.18653/v1/2026.gem-main.34
- Original
- Read it at arXiv
- Topics
Abstract
With the growing need for evaluating Large Language Models (LLMs) and their applications to speech, challenges persist in summarizing and evaluating conversations that lack a clear end goal. We introduce SAUCE -- a reference-free, fact-based evaluation pipeline for cross-lingual conversational speech summarization. It measures the accuracy and the fact coverage of a summary through the entailment between conversation and text. We compare SAUCE against several popular summarization metrics and demonstrate the effectiveness of capturing information loss due to transcription and translation error and identifying broken summaries. Crucially, unlike black-box LLM evaluators or dense embedding metrics, SAUCE is inherently explainable: it maps summary scores to discrete, verifiable facts, allowing users to pinpoint exact hallucinations or omissions. We illustrate how this interpretability helps developers systematically profile LLM behaviors and gives end-users an actionable tool to verify summary accuracy in noisy, real-world conditions. Preliminary investigations show SAUCE strongly align with human judgment.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
ReListen: Training-Free Evidence Localization in Hour-Long Audio with Frozen Audio Language Models
Spatial Transfusion: From Structured Spatial Scene Understanding to Generation and Editing
Deployment-constrained key estimation with a DSP-guided transformer reranker
InteractionBench: Measuring Timing, Task Accuracy, Whisper and Volume, and Grounding in Real-Time Voice Systems
VioTech: Frame-Level Violin Playing Technique Detection with Multi-Scale Mixture-of-Experts
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.