arXiv preprint ·
All-For-One: Distilling Visual Foundation Models to a Fast, Memory-Efficient Visuomotor Policy
- Published
- Source
- arXiv preprint, posted before peer review
- Original
- Read it at arXiv
- Topics
Abstract
Visual Foundation Models (VFMs) provide semantic cues such as depth and segmentation useful for robotic manipulation, but directly deploying them in closed-loop control is impractical due to high computational cost, temporal inconsistency, and distribution shift between static internet images and egocentric robot observations. We propose All-For-One (AFO), a two-stage distillation framework that leverages VFMs strictly as privileged information during training, using downstream robotic data. Following the privileged learning paradigm, AFO decouples semantic representation learning from inference-time policy execution. In the All stage, a teacher policy fuses RGB observations with outputs from multiple VFMs to learn a semantically enriched representation. In the For-One stage, this knowledge is distilled in-distribution into a lightweight RGB-only student policy. Evaluated on two Adroit tasks and five MetaWorld tasks in simulation, as well as three real-world manipulation tasks, AFO outperforms prior foundation-model-based approaches while using over 30x fewer parameters than VFM-based encoders and 50x lower latency than its own teacher in simulation benchmarks, providing an efficient pathway for transferring foundation-model knowledge to robotic control.
The authors’ abstract, as arXiv lists it. arXiv’s listings, abstracts included, are free for anyone to reuse.
Read it on ReadSearch
Each opens after you create an account, and brings you back to this paper.
More in Robotics
VARP: A Visual Adaptive Rearrangement Planner for Non-Monotone Object Rearrangement Planning
Closed-Loop Adaptive Sampling on a Generative Ocean Model
Online Hamilton-Jacobi Reachability Onboard the Robot Control Loop: A Real-Time GPU Solver for Safety-Critical Autonomy
ZipDrift: Generating Multimodal Short-Horizon MPC Costs with One-Step Drifting Models
Say the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAs
Get the few papers worth your time in Robotics every morning.
ReadSearch reads the new papers in your field each day and brings you the few worth reading, each with its key finding in a sentence.