arXiv preprint ·
ERCLIP: Small-Data Adaptation of CLIP for Free-Length Image-Text Retrieval
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
CLIP's 77-token text encoder truncates detailed image descriptions. We present ERCLIP, a small-data adaptation of CLIP for long-caption image--text retrieval. During training, a long caption is split into sentences that the CLIP text encoder encodes independently; the sentence embeddings are fused by mean pooling or a gated residual Transformer scorer and receive their own contrastive loss. At inference ERCLIP is an unmodified CLIP dual encoder: test captions are still read through the 77-token window, and only training sees the full caption. We find that this fine-tuning erodes CLIP's zero-shot classification, most when the fusion output is not a combination of CLIP embeddings, and add a frozen-CLIP preservation loss on the image and short-caption embeddings. With 20k pairs and one epoch, ERCLIP improves long-caption R@1 over CLIP by 6.5--11.2 points, closing 62--77% of the gap to LongCLIP with 2% of its long captions. Preservation recovers 1.5--2.1 ImageNet zero-shot points (30--42% of the loss) and adds up to 2.4 points of COCO image-to-text R@1, at no inference cost.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Information Retrieval
Our Own Jev: Typed Decisions in One Forward Pass, from an In-House Judge to Open LoRA Models
Better Reasoning, Not Just Bigger Models: A Structure-Grounded Framework for Efficient and Reliable Inference
Progressive Learning from Content to Relevance to Personalization for CTR Prediction
The Fusion Tax: A Diagnostic Analysis of When Hybrid Retrieval Hurts
Reasoning-Search-Augmented LLMs: An Operational Taxonomy, Benchmark Analysis and Controlled Evidence Synthesis
Get the few papers worth your time in Information Retrieval every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.