arXiv preprint ·
A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
When two texts describe the same expression, standard metrics based on lexical overlap or whole-text similarity may fail to detect meaningful differences in how that expression is framed. We propose a framework to evaluate semiotic alignment between texts, where a semiotic profile encompasses both the contextual meaning and the discourse references made salient by a text. Our approach yields two scores, Semiotic Fidelity and Semiotic Coverage, estimating how much of one text's profile is supported by the other and how much of the other's profile it recovers. Experiments show that coverage is typically lower than fidelity, and that alignment between LLMs and human-curated data is highest at low sampling temperatures, while higher temperatures reduce this alignment.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Matryoshka attribution: Learning to attribute language model outputs to representations and weights
Realize What Matters: Principled Context Representation for Large-Scale Reasoning
Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure
Rewired or Gated? How Instruction Tuning Shapes Knowledge-Conflict Circuits in LLMs
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.