arXiv preprint ·
Scaling Laws For Data-Constrained Dense Retrieval
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Dense retrieval models are typically obtained by fine-tuning pretrained language models on supervised retrieval data, where high-quality annotations or interaction logs are often limited and reused across many epochs. This data-constrained regime differs from the abundant-data setting assumed by standard scaling laws, making it unclear how the best attainable retrieval performance changes with model size and unique fine-tuning data. We study this question through the minimum achievable evaluation loss (MAEL), defined as the lowest evaluation loss reached during training under fixed model-size and data-budget constraints. We first show that learning-rate choices strongly affect optimization trajectories but have little effect on the final attainable MAEL: after rescaling the optimization-step axis, learning curves collapse across learning rates, and CKA analysis shows that checkpoints reaching MAEL learn highly similar representations. We then fine-tune 35 BERT-family dense retrievers on MS MARCO subsets and find that MAEL is accurately described by a simple multiplicative power law over non-embedding parameters and unique samples, outperforming commonly used additive scaling formulations. We further derive a scaling law for the training step at which MAEL is reached and show that optimal weight decay follows an approximately invariant EMA timescale, yielding lower MAEL and stronger retrieval performance on both in-domain and out-of-domain benchmarks. Finally, combining loss and step scaling suggests that, in our observed regime, increasing pretrained model size is approximately 1.46 times as fine-tuning-compute-efficient as increasing unique in-domain data for reducing MAEL.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Retrieval
IDs That Mean Something: A Hands-On Tutorial on Semantic IDs for Recommender Systems
Item Representations Matter: Bridging Hard Semantic IDs and Soft Prototype Mixtures for Generative Retrieval
IntentGEO: Intent-aware Generative Engine Optimization for Recommendation
SciToT: Multimodal Tip-of-the-Tongue Retrieval for Scientific Papers
ReuseGR: Offline--Online Decoupled Generative Retrieval via Adaptive Prefix Reuse
Get the few papers worth your time in Information Retrieval every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.