Skip to content

LEEVLA: Seeing What Matters in Latent Environment Evolution for Vision-Language-Action

๐Ÿ•’ Published (v1): 2026-07-09 07:28 UTC ยท Source: Arxiv ยท link

Why this paper was selected

Selective visual token routing in VLA; addresses core dynamic-scene limitation in robot VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

LEEVLA is a Vision-Language-Action (VLA) training framework that teaches robot policies to attend to task-critical visual regions and reason about how those regions evolve in latent space. It introduces two training-only modules โ€” drift-guided dynamic prioritization (DGDP) and structured feature flow generation (SFFG) โ€” that impose no additional inference cost while consistently improving performance on manipulation benchmarks.

Problem

Existing VLA models treat all visual patch tokens uniformly during training, diluting supervision across static backgrounds and task-irrelevant regions. Methods that add explicit reasoning (e.g., subgoal image prediction, segmentation, depth) depend on human-selected context clues and auxiliary models, restricting the policy to known causal factors and missing latent, unknown but task-relevant factors. This makes them fragile across viewpoints, lighting, and object appearance variation.

Method

LEEVLA adds two auxiliary training objectives on top of a standard VLA backbone (OpenVLA-7B / miniVLA):

Drift-Guided Dynamic Prioritization (DGDP) computes a per-patch loss weight combining: - Dynamic Position Prioritization (DPP): dynamism score \(\theta_i = 1 - \cos(v_{t,i},\, v_{t+T,i})\), rewarding patches whose features change significantly between timesteps \(t\) and \(t{+}T\). - Semantic Drift Guidance (SDG): instruction-relevance score \(r_{t,i} = \max_{1 \le j \le M} \langle \tilde{x}_{t,i},\, \tilde{c}_j \rangle\), and drift \(\Delta_i = \text{clip}\!\left(\tfrac{r_{t+T,i} - r_{t,i}}{\tau},\,-\delta,\,\delta\right)\), rewarding patches whose semantics shift toward the instruction. The two signals are fused as \(\beta_i = \sigma(\omega_i \cdot \theta_i)\).

Structured Feature Flow Generation (SFFG) specifies how attended features should evolve: - Prototype-to-Periphery (P2P) prediction: clusters multi-view future features into \(L\) clusters; within each cluster, tokens are sorted by Euclidean distance to the centroid. A future-feature decoder predicts \(\hat{v}_{t+T}\), and the P2P loss \(L_{\text{P2P}}\) weights prediction errors by \(\alpha + \beta_j\) per token, prioritizing cluster cores. - Mutual-Neighborhood Contrastive (MC) loss: builds a second-order \(k\)-NN graph on future features and restricts positives to mutual neighbors (both \(j \in G_i^{(1\cup2)}\) and \(i \in G_j^{(1\cup2)}\)), then applies InfoNCE to pull symmetric, high-confidence feature pairs together and push apart spurious neighbors.

Total loss: \(L_{\text{total}} = \lambda_1 L_{\text{action}} + \lambda_2 L_{\text{P2P}} + \lambda_3 L_{\text{MC}}\), where \(L_{\text{action}}\) is L1 action regression. DGDP and SFFG are active only during training.

Key Contributions

  • DGDP: automatic discovery of salient, instruction-relevant regions via coupled spatial dynamism and semantic-drift signals; replaces uniform token weighting with adaptive per-patch loss weights.
  • SFFG / P2P: structured latent-state forecasting that predicts future visual features in prototype-to-periphery order, preserving intra-cluster spatial topology.
  • MC loss: mutual-neighborhood filtering for contrastive supervision robust to noisy robot demonstration data.
  • Zero inference overhead: all auxiliary modules are training-time only, leaving the inference token stream unchanged.

Results

  • CALVIN ABCโ†’D (large-scale, 1โ€“5 step rollout success rates): LEEVLA-large achieves 98.8 / 94.5 / 87.3 / 80.6 / 72.7, avg. chain length 4.34, vs. OpenVLA-OFT 96.3 / 89.1 / 82.4 / 75.8 / 66.9 (avg. 4.10), ฯ€0.5 91.9/84.6/79.4/75.5/71.0 (avg. 4.02), UniVLA 95.5/85.5/75.4/66.9/56.5 (avg. 3.80).
  • LIBERO Object (small-scale, โ‰ค1B params): LEEVLA-mini (0.5B) achieves 98.6% vs. DreamVLA 97.5%, FLOWER 97.1%, UniACT 77.0%, Octo 78.9%.
  • Real-world UR5 (3 tasks, 20 trials each): LEEVLA consistently outperforms OpenVLA across place-object, press-button, and close-drawer tasks (exact numbers in Table 3, not fully reproduced in the provided text).

Limitations

  • Requires temporally paired observation frames at \(t\) and \(t{+}T\) during training; datasets lacking temporal continuity cannot be directly used.
  • P2P clustering introduces additional training-time compute and hyperparameters (\(L\), \(K\), \(M\), \(\lambda_{1{-}3}\), \(\tau\), \(\delta\)).
  • Real-world evaluation is narrow: one robot (UR5), three task types, 20 trials per condition โ€” generalization to diverse embodiments is unverified.
  • The future-feature decoder architecture and most hyperparameter tables are relegated to the supplementary material, limiting reproducibility from the main paper alone.

Relevance to Vision-Language Models

LEEVLA directly addresses a core VLM challenge โ€” grounding visual representations to language under distribution shift โ€” by replacing human-specified attention priors with automatic, loss-level prioritization derived from semantic drift between visual tokens and language embeddings. The SFFG mechanism is a form of self-supervised visual representation learning (akin to masked autoencoders or contrastive methods) applied to the embodied action setting, suggesting that structured latent-space objectives can augment frozen VLM backbones without changing their architecture. For researchers tracking VLMs, the paper demonstrates that token-level weighting strategies and latent-topology constraints are complementary to scaling and prompt engineering, and that these techniques transfer well from discriminative VLM pretraining to the closed-loop robot control domain.