arXiv preprint ·
MoMaTune: Model, Match, Then Tune Language Models for Numerical Records
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Despite remarkable progress in language understanding and generation, decoder language models continue to exhibit striking failures on numerical data. Numerical examples are typically serialized as text by concatenating feature names with their values (e.g., fat:3.4), while conventional task adaptation applies loss only to the downstream prediction target. This leaves two structures weakly supervised: within-value sensitivity, where digit position determines numerical magnitude, because token cross-entropy contains no explicit numerical metric; and across-field structure, where measurements in the same record are statistically dependent, because target-only tuning provides no direct loss on the measurements or their dependencies. We introduce MoMaTune (Model, Match, then Tune), a two-stage post-training strategy organized around these supervision gaps. Stage 1 applies Model, ordinary next-token prediction densely over label-free serialized records, and optionally adds Match, a same-record consistency objective that aligns representations of different partial views of the same example. Stage 2 applies Tune, supervised fine-tuning for downstream target prediction. We evaluate MoMaTune on 169,285 records from eight public datasets, with 339 of 340 input fields numerical. At 1.5B parameters, Model raises the eight-dataset masked-value imputation aggregate from 0.106 to 0.401, while the three-dataset correct-context gain rises from 0.058 to 0.167. On diagnostics, Model also increases sensitivity to numerically consequential digit changes, while Match improves robustness to informative partial views. Adding Tune yields higher target-supervised embedding aggregates for each tested Stage-1 parent, and the two-stage recipes obtain higher aggregate scores than Tune alone. The same pattern is observed at 4B, 9.6B, and 27B parameters when model embeddings are used for downstream prediction.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Artificial Intelligence
Computational Reflective Equilibrium for Agentic AI: A Rawlsian Alignment Scaffold for Governance
Framework for an AI-Ready ARPES Community Database
When Does Compensating Inherited Quantization Error Help? Mean-Square Guarantees for Layer-Wise Quantization and Their Limits
ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail
Of Errors and Echoes: Responsibility-Aware Credit Assignment for Tool-Use Agents
Get the few papers worth your time in Artificial Intelligence every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.