Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgetting

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Fine-tuning VLMs into VLAs without catastrophic forgetting; solves core challenge

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLM2VLA eliminates catastrophic forgetting in VLA fine-tuning by representing low-level robot actions as natural language text rather than discrete tokens or continuous vectors, aligning the robotics fine-tuning distribution with the VLM's pretraining corpus. This alignment makes LoRA-only fine-tuning sufficient — no co-training on internet-scale data required — while retaining over 85% of the base VLM's multimodal reasoning capability and enabling zero-shot generalization to novel tasks.

Problem

Fine-tuning pretrained VLMs on robot teleoperation data (creating VLAs) causes catastrophic forgetting: the VLM's semantic reasoning, instruction following, and generalization to novel objects degrade because robot action spaces (continuous vectors or arbitrarily assigned discrete tokens) are severely out-of-distribution relative to the image-text pretraining corpus. Existing mitigations either require expensive co-training with web-scale VLM data or complex architectural changes (MoE stop-gradients, frozen backbones + separate action heads), all of which add significant engineering burden.

Method

The core insight is to resolve the distribution mismatch at the data level rather than the model level. VLM2VLA re-represents each robot teleoperation trajectory as a hierarchical natural language description via a three-stage factorization:

\[p_\theta(\bar{a}_i, m_i, l_i \mid \bar{o}_i, L) = p_\theta(l_i \mid \bar{o}_i, L)\; p_\theta(m_i \mid l_i, \bar{o}_i)\; p_\theta(\bar{a}_i \mid m_i, l_i, \bar{o}_i)\]

1. Subtask prediction \(l_i\): high-level description of the immediate sub-goal.
2. Motion planning \(m_i\): directional end-effector description (e.g., "move down and slightly forward").
3. Action generation \(\bar{a}_i\): variable-length action chunk expressed as text (e.g., [0.1, 0.03, -0.2, 1] described as "move down toward the pepper and grasp it").

Gemini 2.5 automatically re-labels the BridgeV2 teleoperation dataset \(D_\text{rob}\) into language-annotated trajectories \(D_\text{lan}\). The VLM (Gemma-3-12B-IT) is then fine-tuned solely with LoRA applied to all linear modules using cross-entropy loss on this natural language corpus, casting robot control as standard supervised next-token prediction. A Gemini 2.5 Pro verifier operates closed-loop at test time to determine whether to retry the current subtask or advance to the next.

Key Contributions

• Actions as language representation: translates low-level numerical actions into text strings within the VLM's existing vocabulary, shrinking the distribution gap between \(D_\text{lan}\) and the pretraining corpus.
• LoRA-only VLA training pipeline: demonstrates that parameter-efficient fine-tuning is sufficient for robot control when the data distribution is properly aligned, eliminating the need for co-training or architectural surgery.
• Automated data curation pipeline: scalable Gemini-based annotation scheme to convert any teleoperation dataset into hierarchical natural language format.
• Empirical validation at scale: over 800 real-world robot manipulation experiments plus comprehensive VQA benchmark evaluation, with ablation of action token vs. action language representations.

Results

• VQA retention: VLM2VLA retains >85% of Gemma-3-12B-IT base performance across 12 VQA benchmarks (MMMU, MMStar, MME, OCRBench, MMB-en/cn, TextVQA, DocVQA, InfoVQA, AI2D, ChartQA, RealWorldQA), while OpenVLA and ECoT drop to near-zero on most.
• VQA vs. co-trained baselines: VLM2VLA outperforms MolmoAct (co-trained) across all reported benchmarks; e.g., MMStar 48.0 vs. 1.2, MME 1391.7 vs. 1224.5, TextVQA 68.5 vs. 55.1.
• VLM2VLA-AT ablation: action-token variant achieves comparable VQA scores (e.g., MMMU 45.9 vs. 42.7), confirming LoRA is the primary mechanism for VQA retention, but language actions improve downstream robotic generalization.
• Real-world robotics: superior out-of-distribution generalization over OpenVLA and ECoT on novel objects and multilingual instructions across 800+ experiments (exact per-task numbers in appendix/truncated sections).

Limitations

• Only translational degrees of freedom (DoF) were considered in this work; rotational DoFs were not evaluated.
• Relies on Gemini 2.5 as both the trajectory annotator and closed-loop verifier, introducing a dependency on a proprietary external model for both data curation and inference.
• The data curation pipeline assumes access to main-task language instructions in the source dataset; trajectories without instructions require additional processing.
• Evaluated exclusively on BridgeV2 data and a single robot platform; breadth of embodiment generalization is untested.
• Subtask plans are generated from the initial observation \(\bar{o}_0\) and held fixed throughout the rollout, which may be brittle under large environmental perturbations.

Relevance to Vision-Language Models

VLM2VLA directly addresses a core tension in adapting VLMs for embodied AI: how to leverage internet-scale multimodal pretraining without destroying it during task-specific fine-tuning. By framing this as a representational alignment problem rather than an architectural one, the work challenges the widespread assumption that co-training on VLM-scale datasets is necessary to prevent catastrophic forgetting — a result with broad implications for parameter-efficient adaptation of VLMs in any grounded domain. The finding that LoRA is sufficient when the data distribution matches clarifies where the forgetting originates and sets a principled constraint for future VLA dataset design. For VLM researchers, the systematic VQA benchmarking protocol (12 benchmarks, comparison against state-of-the-art co-trained models) offers a reusable evaluation template for measuring knowledge retention after domain fine-tuning.