arXiv preprint ·
Reducing Representational Interference in Retrieval Embeddings with Trainable Prefix Tokens
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Embedding models compress variable-length text into fixed-size vector representations, but this compression can entangle semantic content with residual token-level information, such as artifacts arising from attention sink behavior. We propose a simple method that prepends trainable [DUMP] tokens to the input during embedding model training. The method keeps the model architecture, pooling strategy, and training loss unchanged, while optimizing the added tokens through the standard contrastive retrieval objective. By introducing trainable prefix positions that interact with regular tokens during encoding, [DUMP] tokens help reduce the influence of residual or uninformative token-level signals on the final retrieval representation. Experiments on standard retrieval and long-context benchmarks show that our method consistently improves over finetuned baselines across encoder and decoder embedding models. Further analysis shows that [DUMP] tokens alter token-level activation and attention patterns associated with low-information tokens. These results suggest that trainable prefix tokens provide a lightweight way to improve retrieval representations.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Retrieval
CacheGraft: Enhancing Retrieval-Augmented Generation via Latent-Space Document Enrichment
BRIDGE: Bridging the Semantic Quantization Gap for Generative Recommendation
MEvince: Knowledge-Constrained LLM Reasoning for Technique Identification from Malware Reports
OptST: Optimizing Semantic Tokenization for LLM-Enhanced Recommendation
What Gets Hacked in a Collaborative Reward for LLM Recommenders: The Part That Does Not Depend on the User
Get the few papers worth your time in Information Retrieval every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.