Not All Layers Need Tuning: Diagnosing and Directing Adaptation in Vision-Language-Action Models

Abstract

Fine-tuning a Vision-Language-Action (VLA) model for a new deployment environment is expensive, yet most methods apply uniform-capacity adapters to every network region as if every region requires equal adjustment. This paper tests that assumption on five architecturally diverse VLAs (OpenVLA-OFT, $π_0$, SmolVLA, DTP, Octo; 93M-7B parameters). Measuring per-region adaptation cost as normalized parameter displacement under region-isolated fine-tuning reveals an adaptation spectrum in which appearance shifts concentrate cost in the vision encoder, instruction shifts in the language backbone, and novel-object shifts in the vision encoder together with the action head, across all five architectures. To exploit this structure, we introduce a pipeline that observes, diagnoses, allocates, and adapts. From ten unlabeled target observations and without fine-tuning, the diagnostic estimates per-region cost by combining reference-free gradient and Monte Carlo Dropout signals with a Centered Kernel Alignment score against a cached source reference; the allocator converts the estimates into variable-rank LoRA adapters under a parameter budget and freezes well-calibrated regions; and standard LoRA fine-tuning trains the resulting adapters. The diagnostic ranks regions within each deployment at a median Spearman of 0.91, and the allocation matches or exceeds uniform LoRA at every budget we tested on LIBERO and CALVIN. On a physical xArm-7, the pipeline matches full fine-tuning under an instruction-wording shift with 0.04% of its trainable parameters, and on five held-out scenes evaluated without retraining it leads every baseline, with 11-23 successes of 30 rollouts against 8-18 for the strongest parameter-efficient baseline at equal or larger budgets and 2-11 for full fine-tuning. These results suggest that adaptation cost in VLAs is structured enough to measure before fine-tuning begins.

The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.

More in Robotics

• arXiv preprint · 16 September 2026

  Feedback-Modulated Harmonic Policies for Quadruped Locomotion

  Yixuan Jia, Steven Roche and Jonathan P. How
• arXiv preprint · 17 September 2026

  CoreSense: Traceable Failure Recall and Conflict-Aware Belief Gating for Auditable Robot Decisions

  Zoe Li
• arXiv preprint · 16 September 2026

  Missing Bridges: Composition-Aware Active Imitation Learning

  Maxwell J. Jacobson, Ahmed H Qureshi and Yexiang Xue
• arXiv preprint · 17 September 2026

  AMB3R-SLAM: Kilometer-scale SLAM with Hierarchical Backend

  Hengyi Wang and Lourdes Agapito
• arXiv preprint · 16 September 2026

  Causal-History Test-Time Scaling for Failure Recovery in Autoregressive World-Action Models

  Lin Li, Long Chen, Kwunhang and 10 others

All new papers in Robotics