Skip to content

VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLA-OS is a unified Vision-Language-Action model family that enables controlled, architecture-agnostic comparison of task planning paradigms (ActionOnly, Integrated, Hierarchical) and representations (language, visual, image foresight) for robot manipulation. Through exhaustive ablations across six benchmarks, it finds that visually grounded planning representations outperform language-based ones, and that Hierarchical-VLA generally achieves best performance at the cost of slower training/inference. Policy learning—not task planning—is the primary bottleneck for current manipulation tasks.

Problem

Existing VLA approaches that incorporate task planning differ across network architectures, planning paradigms, representation types, and training data sources simultaneously, making it impossible to attribute performance gains to specific design choices. No systematic, controlled study existed to answer: which planning representation works best, whether monolithic vs. hierarchical planning is better, where the performance bottleneck lies, and whether scalability/pretraining benefits transfer to planning-augmented VLAs.

Method

The authors build VLA-OS—three composable model variants sharing a unified VLM backbone (Qwen2.5, 0.5B–7B) with plug-and-play planning and action heads: - VLA-OS-A (ActionOnly): block-wise causal attention VLA with DINOV2+SigLIP vision encoder ensemble, flow-matching or L1 action head. - VLA-OS-I (Integrated): adds language, visual (location-token-based bounding boxes/end-effector flow/affordance), or image foresight (VAR-style autoregressive goal image) planning heads with both implicit (auxiliary loss only) and explicit (action head attends to planning head KV) variants. - VLA-OS-H (Hierarchical): separate VLM+planning head for high-level planning and an encoder-decoder action head that ingests raw observations plus the planning output; no gradient coupling between the two.

All planning heads have ~5% of VLM parameter count to equalize comparisons. Experiments span LIBERO (2D rigid), COLOSSEUM (3D/generalization), FurnitureBench (long-horizon), real-world deformable tasks, DexArt (dexterous hands), and PerAct2 (dual-arm). Separate Decomposition Score (DCS) and Instruction Following Score (IFS) metrics disentangle planning accuracy from policy execution quality.

Key Contributions

  • Unified VLA-OS model series enabling fair, architecture-controlled comparison across three planning paradigms and three representation types.
  • Annotated task reasoning dataset covering language reasoning (8 structured keys), visual reasoning (location tokens for boxes/flow/affordance), and image foresight (future frame).
  • 12 empirical findings from controlled ablations across diversity of tasks, modalities, scales, and generalization settings.
  • Open-source release of code, annotated datasets, and checkpoints.

Results

  • Sanity check: VLA-OS-A-S (0.5B, scratch) achieves 85.6% average on LIBERO vs. Diffusion Policy 72.4%, OpenVLA 76.5%, CoT-VLA 81.1%, Ï€0-FAST 85.5% (fine-tuned); below SOTA Ï€0 94.2% (fine-tuned).
  • Representation (LIBERO-Long): Best implicit Integrated gains: IF +6.5%, V +5.0%, L +2.0%. Explicit planning degrades severely (L: −5.5%, V: −13.5%); image foresight partially survives (+1.5%). Hierarchical best single representation: V+IF+L combined 74.2% vs. ActionOnly 66.0%.
  • Paradigm breadth: Integrated (73.3%) and Hierarchical (74.2%) both outperform ActionOnly (66.0%) on LIBERO-Long; gains consistent across COLOSSEUM 3D, deformable real-world, DexArt, PerAct2.
  • Generalization (COLOSSEUM ALL-Perturbation): Hierarchical 7.4 vs. Integrated 6.2 vs. ActionOnly 6.1.
  • Planning head pretraining: Both Integrated (79.1%) and Hierarchical (79.8%) gain ~5.6–5.8 pp over ActionOnly baseline on LIBERO-Long after pretraining on LIBERO-90.
  • Separate evaluation: IFS consistently lower than DCS across all representations, confirming policy learning is the harder bottleneck.
  • Scalability: With ~5,000 demonstrations, performance peaks at 0.5B LLM; larger backbones hurt. All paradigms scale with data.

Limitations

  • Several planning variants not covered: latent-action-based image generation, video-generation planning, scene-flow planning.
  • No embodiment transfer, sim-to-real transfer, or 2D-to-3D transfer study.
  • Training data capped at <10,000 trajectories; large-scale pretraining datasets (e.g., OXE) not explored.
  • Explicit Integrated-VLA suffers severe planning accumulation error—not fully solved, only diagnosed.
  • Inference and training speed of Hierarchical-VLA remain significantly higher than alternatives.

Relevance to Vision-Language Models

VLA-OS directly probes how VLMs should be adapted for embodied action generation, making it highly relevant to anyone tracking VLM grounding, compositionality, and chain-of-thought reasoning in physical domains. The finding that visually grounded representations (location tokens, image foresight) outperform language chains-of-thought challenges the dominant text-centric reasoning paradigm in VLMs and suggests that spatially tokenized visual outputs are more actionable for downstream motor control. The result that explicit planning in a monolithic model degrades performance due to accumulation error—while hierarchical decoupling mitigates it—has direct implications for how VLMs should be coupled to action decoders in future embodied AI architectures.