arXiv preprint ·
Reversion, Retention, and Semantic Intervention: Measuring Functional Content in CLIP Fine-Tuning Deltas
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Fine-tuning adapts pretrained models to new tasks, but identifying which weight changes support the resulting behavior remains difficult. Weight-space interpretability assigns semantic descriptions to these changes, called fine-tuning deltas. Such descriptions leave open how much task performance depends on the identified updates, particularly when other updates can support the same predictions. We address this gap by pairing reversion, which restores selected parameters to their pretrained values, with retention, which keeps only selected fine-tuning updates. We evaluate Contrastive Language-Image Pretraining (CLIP) models on synthetic financial charts, Flowers102, EuroSAT, and FGVCAircraft with ViT-L/14, and on Flowers102 with ViT-B/16, using a shared recipe that tunes the final six visual blocks. Our contributions are threefold. First, attention value-output (VO) updates alone recover 43-87% of the accuracy improvement, although reverting them leaves chart accuracy unchanged. Cross-entropy loss preserves this contrast on Charts and EuroSAT, while correct-class score margins reveal additional contributions. Second, VO weights recover more loss improvement than all ten equally sized, layer-matched random subsets in each setting. VO and multilayer perceptron (MLP) retentions jointly correct 25-81% of pretrained errors, demonstrating overlapping recovered behavior, which we call operational redundancy. Third, removing semantically selected directions reduces VO-retained recovery more than matched-random removal in all five settings. Together, these results distinguish semantic alignment, removal cost, and retained recovery as complementary evidence about the functional role of fine-tuning updates within the evaluated CLIP recipe.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Computer Vision
Continual Learning Exam: Benchmarking Long-Horizon RSI and Experiential Learning in LLM Agents
Regional prediction of hematoma expansion from non-contrast computed tomography in spontaneous intracerebral hemorrhage patients
Accuracy Is Not Enough: An Efficiency- and Statistics-Aware Benchmark of ResNet50 and Vision Transformer for Explainable Multi-Class Gastrointestinal Diseases Classification
StreamVibe: Proactive Video Agents for Evolving User Intent
InteractiveWorld: A Verifiable Human-Centric Environment for Building Better Interactive MLLM Assistants
Get the few papers worth your time in Computer Vision every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.