Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignment¶

🕒 Published (v1): 2026-07-15 04:13 UTC · Source: Arxiv · link

Why this paper was selected

Representation anchoring + language-action alignment; fixes BC fine-tuning catastrophic forgetting in VLAs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Behavior cloning finetuning of VLMs into robot policies progressively erases pretrained visual-semantic representations and leaves language and action heads misaligned. Anchor-Align addresses both failures by adding a layer-wise distillation loss from a frozen VLM copy and a programmatic language-action alignment loss derived from motion-direction labels. The method improves real-robot success rates from 28%→54% and 37%→60%, and consistently outperforms baselines across OOD simulation benchmarks.

Problem¶

Standard BC finetuning of VLMs into VLA policies has two failure modes: (1) catastrophic forgetting—action-loss gradients overwrite visuolinguistic representations acquired at internet scale, eliminating color/spatial grounding; and (2) language-action misalignment—co-training on generic VQA/captioning data supervises language and action heads on disjoint observations, allowing the two heads to make contradictory predictions (e.g., language predicts "left" while the action head moves right). Existing co-training remedies score 0% on position-swap generalization tests, demonstrating they do not sufficiently constrain backbone representations.

Method¶

Vision-Language Anchoring: A frozen copy of the pretrained VLM (the "anchor") processes the same input batch in parallel. At every decoder layer \(u \in D\), a squared Frobenius norm loss penalizes drift between trainable backbone hidden states \(H^S_u\) and anchor hidden states \(H^A_u\) for all vision and text token positions \(m\):

\[\mathcal{L}_{\text{anchor}} = \frac{1}{|D|} \sum_{u \in D} \|H^S_u[m] - H^A_u[m]\|^2_F\]

Language-Action Alignment: For each non-stationary training chunk, the continuous action target \(a_{\text{cont}} \in \mathbb{R}^{B \times K \times 7}\) is converted to a discrete direction label \(a_{\text{lang}} \in \{\text{up, down, left, right, forward, backward}\}\) by averaging the translational components across the chunk, filtering near-stationary samples by \(\|\bar{v}_i\|_2 < \tau\), and selecting the dominant axis sign. The pre-action hidden state of the last instruction token \(h_{\text{pre}} = H^S[N] \in \mathbb{R}^d\) is projected through a learned \(W_{\text{proj}}\) and the frozen LM head \(W_{\text{lm}}\) to produce logits \(\hat{a}_{\text{lang}}\), trained with cross-entropy:

\[\mathcal{L}_{\text{align}} = \text{CE}(\hat{a}_{\text{lang}},\, a_{\text{lang}})\]

The total loss is \(\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{action}} + \lambda_{\text{anchor}}\mathcal{L}_{\text{anchor}} + \lambda_{\text{align}}\mathcal{L}_{\text{align}}\). The method requires no additional data, no architectural changes beyond the frozen copy, and is evaluated on two base architectures: VLA-Adapter (regression head) and StarVLA (flow-matching GR00T FM-DiT head).

Key Contributions¶

Vision-Language Anchoring: layer-wise representation distillation from a frozen pretrained VLM, preventing catastrophic forgetting without auxiliary datasets.
Language-Action Alignment: automatic, annotation-free conversion of continuous action trajectories into discrete motion-direction language labels to co-supervise both VLM and action heads on the same robot observation.
Diagnosis of language-action misalignment in co-trained VLAs—first direct empirical demonstration that co-trained language and action heads contradict each other on robot observations.
Regularization controls (Shuffle/Scatter) confirming that gains arise from genuine grounded alignment, not auxiliary task regularization.
Consistent improvements across two VLA architectures (regression and flow-matching), three simulation benchmarks (LIBERO-PRO, LIBERO-Plus, CALVIN ABC→D), and real xArm7 experiments.

Results¶

Real robot (xArm7): VLA-Adapter backbone 28%→54%; StarVLA backbone 37%→60% task success.
LIBERO-PRO mean success: VLA-Adapter 61.0% → Anchor-Align 71.9%; position-swap (hardest axis) 2.3%→22.6% vs. 0% for all co-trained baselines.
LIBERO-Plus mean success: 85.1%→90.3%; largest per-axis gains: background texture +8.9%, sensor noise +7.4%, robot initial state +6.5%.
CALVIN ABC→D: average chain length 4.3→4.5; five-instruction completion 73.1%→77.9% (vs. OpenVLA-OFT 66.5%); gains widen with chain depth (+0.8 at k=1 to +4.8 at k=5).
Ablation: anchoring-only (68.1% PRO) and alignment-only (65.9% PRO) each improve over BC baseline (61.0%); combined reaches 71.9%.
Co-training + Knowledge Insulation baseline scores 0% on LIBERO-PRO position-swap, confirming auxiliary data co-training is insufficient.

Limitations¶

Position-swap remains hard: Anchor-Align VLA reaches only 22.6% (up from 2.3%), indicating residual failure to generalize to fully rearranged scenes.
Direction label vocabulary is coarse (six axis-aligned words), losing rotational and multi-axis motion semantics.
Requires maintaining a full frozen VLM copy during training, increasing GPU memory proportionally to backbone size.
Evaluated on relatively small VLM backbones (Qwen2.5-0.5B / Qwen2.5-VL); generalization to billion-parameter-scale VLAs is not demonstrated.
Real-world evaluation is limited to a single robot platform (xArm7) and a fixed task suite.

Relevance to Vision-Language Models¶

Anchor-Align directly addresses a fundamental tension in VLM adaptation: how to transfer internet-scale visual-semantic priors into downstream task-specific policies without destroying those priors. The layer-wise representation distillation technique is a practical instantiation of anti-forgetting regularization that extends the teacher-student distillation literature specifically to multimodal decoder-only transformers. For VLM researchers, the language-action alignment component reveals a previously underdiagnosed failure mode—semantic inconsistency between a VLM's language generation head and a grafted action head sharing the same backbone—which has broader implications for any architecture that extends a VLM with a non-language prediction head. The paper's regularization controls (Shuffle/Scatter) provide a reusable diagnostic protocol for distinguishing genuine semantic grounding from auxiliary-task regularization effects in VLM finetuning.