Semantic Anchoring for Robotic Action Representations

🕒 Published (v1): 2026-07-15 08:45 UTC · Source: Arxiv · link

Why this paper was selected

Semantic anchoring preserves VLM priors in VLA fine-tuning; Yizhou Wang (PKU) group

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLA models degrade the rich semantic structure inherited from pretrained VLMs during fine-tuning on narrow robot demonstrations. This paper diagnoses that erosion via alignment probing, then introduces a training-time-only plug-and-play fix: contrastive anchoring of mid-layer action representations to a frozen semantic manifold with shared/private channel decomposition. The auxiliary modules are fully discarded at inference, yielding up to +18.7% in-distribution and +21.5% OOD gains on a real bimanual robot.

Problem

Standard action-only fine-tuning of VLAs on limited demonstration data erodes the intention-level semantic structure inherited from VLM pretraining. This collapse — visible as loss of task-level clustering in mid-layer representations — causes shortcut learning and undermines generalization to out-of-distribution objects, positions, and task compositions. No prior diagnostic framework existed to measure this degradation, and existing remedies (data scaling, architecture redesign) addressed symptoms without identifying the root cause.

Method

Diagnostic (§2): At layer \(k=10\) of \(\pi_0\) (18-layer backbone), per-token action features are mean-pooled and aligned to instruction embeddings from a frozen Qwen3-VL-Embedding encoder using bidirectional InfoNCE:

\[\mathcal{L}_{\text{align}} = -\frac{1}{2B}\sum_{n=1}^{B}\left[\log\frac{\exp(\text{sim}(\hat{z}_a^{(n)}, \hat{z}_t^{(n)})/\tau)}{\sum_m \exp(\text{sim}(\hat{z}_a^{(n)}, \hat{z}_t^{(m)})/\tau)} + \log\frac{\exp(\text{sim}(\hat{z}_t^{(n)}, \hat{z}_a^{(n)})/\tau)}{\sum_m \exp(\text{sim}(\hat{z}_t^{(n)}, \hat{z}_a^{(m)})/\tau)}\right]\]

Bidirectional Recall@1 under lightweight projection heads \(T_a, T_t\) serves as the alignment metric.

Method (§3): At the same mid-layer \(k\), per-token features \(h_i^{(k)}\) are decomposed via MLP encoders into a \(d_s\)-dimensional shared channel \(z_i^s = E_s(h_i^{(k)})\) (intention-level semantics) and a private channel \(z_i^p = E_p(h_i^{(k)})\) (execution-specific detail), with additive reconstruction \(h_i^{\text{rec}} = \text{Dec}(z_i^s + z_i^p)\). Only shared channels enter cross-attention pooling with a learnable query \(q \in \mathbb{R}^{d_s}\); the pooled aggregate is then contrastively aligned to a frozen EgoHOD text encoder embedding. Three auxiliary losses are added:

\[\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{action}} + \lambda_{\text{align}}\mathcal{L}_{\text{align}} + \lambda_r\mathcal{L}_{\text{recon}} + \lambda_d\mathcal{L}_{\text{diff}}\]

with \(\lambda_r=0.01\), \(\lambda_d=0.075\). An angular decorrelation penalty \(\mathcal{L}_{\text{diff}}\) prevents the two channels from re-entangling. All auxiliary modules are discarded at inference.

Key Contributions

• Probing framework demonstrating that action-only fine-tuning progressively erodes VLM-inherited semantic structure, with OOD success synchronizing with alignment quality (Spearman \(\rho=0.964\)) while ID success rises monotonically.
• Evidence that per-trajectory alignment (cosine similarity and retrieval accuracy) predicts individual rollout success/failure at a fixed checkpoint.
• Plug-and-play semantic anchoring method with shared/private decomposition that adds zero inference cost and is backbone-agnostic (validated on both flow-matching \(\pi_0\) and autoregressive SpatialVLA).
• Identification that manipulation-centric video encoders (EgoHOD) outperform general VLMs (Qwen3-VL-Embedding) and static encoders (CLIP-Large) as alignment targets, isolating the contribution of temporal visual dynamics and domain-specific pretraining over scale.

Results

• LIBERO (simulation, \(\pi_0\)): +3.1% average success (89.3% → 92.4%) across four suites with only 50 demonstrations per task.
• SimplerEnv (\(\pi_0\)): +6.3% average (35.4% → 41.7%); SpatialVLA: +7.2% average (43.8% → 51.0%) under real-to-sim domain shift.
• Real bimanual robot (AgileX Cobot Magic V2, \(\pi_0\) baseline):
• In-distribution: 51.3% → 70.0% (+18.7%) across four task families (fruit pick & place, dish stacking, box packing, cabinet storage).
• Out-of-distribution: 49.5% → 71.0% (+21.5%) across five axes (language, position, object, visual, task composition).
• Largest OOD gains on Position (+25pp) and Task (+30pp) axes.
• Ablation (SpatialVLA/SimplerEnv): Contrastive alignment alone: 43.8% → 49.0%; adding shared/private decomposition: 49.0% → 51.0%; EgoHOD target outperforms Qwen3-VL-Embedding (51.0% vs 50.0%) and CLIP-Large (47.5%).

Limitations

• Performance is upper-bounded by the quality of the frozen alignment encoder; a weaker manipulation encoder will limit gains.
• Alignment objective is applied only during post-training on task-specific demonstrations; pre-training integration is unexplored.
• Evaluation of OOD generalization is restricted to the pick-and-place family; broader compositional OOD coverage is absent.
• The \(\lambda_{\text{align}}\) hyperparameter is backbone-dependent (≈0.1 for \(\pi_0\), ≈0.5 for SpatialVLA), requiring per-backbone tuning.

Relevance to Vision-Language Models

This paper directly addresses the catastrophic forgetting of VLM representational structure during VLA fine-tuning — a fundamental bottleneck in transferring pretrained semantic knowledge to embodied policies. The diagnostic finding that mid-layer alignment quality predicts OOD generalization provides a concrete, measurable criterion for evaluating VLM-derived representations in downstream task-specific fine-tuning, applicable beyond robotics to any VLM fine-tuning regime. The shared/private decomposition draws on REPA-style training-time alignment and extends it to resolve the gradient conflict between semantic invariance and task-specific sensitivity. For VLM researchers, the result that manipulation-centric video encoders (EgoHOD) outperform large general VLMs as alignment anchors highlights the importance of domain-appropriate semantic reference spaces even when model scale favors the general encoder.