From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Model¶

🕒 Published (v1): 2026-05-21 16:14 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; learning generalized behavioral representations for VLA distribution shift

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

BehaviorVLA addresses distribution-shift brittleness in Vision-Language-Action (VLA) models by learning a low-dimensional behavioral manifold, factorized into a time-invariant global prototype and a time-variant phase state. A causal three-stream Mamba encoder (VBE) abstracts long-horizon trajectories, while a Phase-conditioned Behavior Decoder (PBD) instantiates them into precise actions via a Predictor-Corrector flow policy. The framework achieves state-of-the-art results on RoboTwin 2.0, LIBERO, and CALVIN, and matches fully fine-tuned OpenVLA-OFT in sim-to-real transfer using only 50% of real-world demonstrations.

Problem¶

Standard VLA models learn observation-to-action mappings directly in high-dimensional ambient space without manifold constraints. Under domain shift (varying lighting, textures, object placement, sim-to-real), predicted actions drift off the valid task manifold. Prior latent-action approaches (VAE-based chunking: ACT, VQ-BeT) suffer from (i) short-horizon temporal fragmentation — trajectory chunks are decoded independently, losing global task coherence — and (ii) static execution-alignment — a fixed latent variable is decoded without awareness of real-time execution progress, causing temporal drift.

Method¶

BehaviorVLA imposes a hierarchical manifold factorization on the policy:

\[p(a_{t:t+k}|O_t, L) = \int p(a_{t:t+k}|z_\text{proto}, z_\text{phase}, O_t, L)\, p(z_\text{phase}|O_t, a_{t-1})\, p(z_\text{proto}|O_0, L)\, dz\]

Visuomotor Behavior Encoder (VBE) — causal three-stream architecture (vision \(S_v\), action \(S_a\), behavior \(S_z\)): - Each stream processes its sequence with a Mamba SSM (linear-complexity \(O(L)\)) using input-dependent timescale \(\Delta_t = \text{Softplus}(\text{Linear}(x_t))\) to selectively filter noise. - Spatial fusion: vision and action streams exchange information via mutual cross-attention; the behavior stream then queries their concatenated keys/values to form an information bottleneck. - Global prototype \(z_\text{proto}\): offline temporal mean-pooling of behavior tokens, stored in a memory bank; at inference, top-\(K\) entries are retrieved via \(\hat{z}_\text{proto} = \sum_{i \in N_K} \text{softmax}(\langle q, k_i\rangle/\kappa)\, z_\text{proto}^{(i)}\). - Phase state \(z_\text{phase}\): recursively updated online from \((O_t, a_{t-1})\) to track execution progress.

Phase-conditioned Behavior Decoder (PBD) — Predictor-Corrector paradigm: - Predictor: unfolds \(\hat{z}_\text{proto}\) into latent anchors \(M \in \mathbb{R}^{H \times D}\); phase state \(z_\text{phase}^{(t)}\) queries \(M\) via Progress-Attention to interpolate a local context \(c_t\), which parameterizes a Gaussian prior \(\mathcal{N}(\mu_\psi(c_t), \text{diag}(\exp(\sigma_\psi(c_t))))\). - Corrector: Conditional Flow Matching policy \(\pi_\theta\) refines this prior via latent structural biasing \(\tilde{e}(a_\sigma) = e(a_\sigma) + \lambda \cdot \text{Proj}_\phi(\mu_\text{prior})\), shifting the flow trajectory toward the prior-defined high-probability region.

Two-stage training: Stage 1 learns the behavior manifold (\(\mathcal{L}_\text{rec}\) via JEPA-style predictive reconstruction + \(\mathcal{L}_\text{global}\) supervised contrastive clustering + \(\mathcal{L}_\text{local}\) InfoNCE for temporal distinctiveness). Stage 2 jointly optimizes flow matching loss and NLL prior loss, with stochastic prior dropout to prevent posterior collapse.

Key Contributions¶

VBE: causal three-stream Mamba architecture decoupling trajectory into time-invariant \(z_\text{proto}\) (scene-invariant topology) and time-variant \(z_\text{phase}\) (progress tracking), enabling long-horizon coherence without quadratic attention cost.
PBD: Progress-Attention interpolation of manifold anchors combined with geometry-guided flow matching, replacing uninformative Gaussian priors with phase-aligned structural priors.
Behavior Memory Bank: offline retrieval of global prototypes at episode start, providing a stable behavioral prior throughout execution without per-step retrieval cost.
State-of-the-art on RoboTwin 2.0 Hard, LIBERO, and CALVIN; 63% average real-world success-rate improvement; 2× data efficiency vs. OpenVLA-OFT in sim-to-real transfer.

Results¶

RoboTwin 2.0 (Hard, 20 tasks, 100 rollouts): 58.0% vs. π0.5 49.2%, π0 24.0%, RDT 42.5% (best prior)
LIBERO (500 rollouts/suite): 98.0% avg (99.2 / 99.4 / 98.8 / 94.6 across Spatial/Object/Goal/Long) vs. π0.5 96.9%, OpenVLA-OFT 97.1%, MemoryVLA 96.7%
CALVIN (ABC→D): 4.36 Avg.Len (full results in appendix)
Real-world (8 tasks): BehaviorVLA at 50% training data matches OpenVLA-OFT at 100%; at 100% data, outperforms both OpenVLA-OFT and π0.5 across all 8 tasks
Ablation (LIBERO): VBE alone: +0.4–1.4pp over base on long-horizon; PBD alone: +1–2pp; both together best on all suites. Real-world gains are larger: VBE adds +8pp generalization, +7pp long-horizon; PBD adds +3/+4pp; combined: +13/+14pp.
Guidance strength \(\lambda\): peak performance at \(\lambda = 0.5\); degradation at extremes.
Memory bank size \(K\): optimal at \(K = 5\) for most LIBERO suites.

Limitations¶

Built on the π0.5 backbone; performance is inherited from and bounded by that pretrained VLM-action-expert architecture.
Memory bank construction requires offline processing of full demonstration trajectories; not applicable in truly zero-shot settings.
Retrieval quality at inference depends on the representativeness of the bank; out-of-distribution tasks with no similar demonstrations in the bank may degrade \(z_\text{proto}\) quality.
CALVIN results are deferred to the appendix in the provided text, preventing direct comparison of that benchmark in the main table.
Real-world evaluation covers only 8 tasks; breadth of manipulation scenarios tested remains limited.

Relevance to Vision-Language Models¶

BehaviorVLA directly extends VLMs into the action domain by augmenting a vision-language backbone (π0.5) with structured latent behavioral representations, showing that VLM features alone are insufficient for robust manipulation under distribution shift — explicit temporal and manifold structure is required. The work advances the line of research connecting VLMs to embodied control (OpenVLA, π0, gr00t-N1) by introducing a principled disentanglement of scene-invariant task topology from progress-specific dynamics, an approach directly applicable to any VLM-based policy. The Behavior Memory Bank mechanism is conceptually related to retrieval-augmented VLMs but operates over sensorimotor trajectories rather than text/image embeddings. Researchers tracking VLMs will find the sim-to-real data-efficiency result (2× reduction) practically significant for deployment.