arXiv preprint ·
Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. We evaluate Jev as a semantic decision component using a harness that follows its documented guidance and assigns arithmetic to code. The study compares twelve model configurations on twenty source-grounded Choices across ten scientific cases, each repeated five times. We measure semantic selections, downstream outputs and final claim labels separately. Jev matched five other configurations at complete semantic correctness and achieved the lowest observed median latency among successful responses. Across three comparison models, seven wrong selections on one culture-history question changed downstream counts while preserving the correct final label. These results identify a useful role for Jev in prepared scientific decision tasks and show why evaluating that role requires checking the relations and quantities that a workflow will reuse.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computation & Language / NLP
Some Dialects Are More Equal Than Others: Non-Prestigious Arabic Dialectal Bias in LLMs
Matryoshka attribution: Learning to attribute language model outputs to representations and weights
Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model
Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference
From Tables to Quantified Statements: Evaluating LLM Inference Generation through Executable Verification
Get the few papers worth your time in Computation & Language / NLP every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.