Fast-in-Slow: A Dual-System VLA Model Unifying Fast Manipulation within Slow Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Fast-in-Slow (FiS-VLA) is a dual-system Vision-Language-Action model that embeds a fast execution module (System 1) directly within the final transformer blocks of a pretrained VLM (System 2), rather than attaching a separate policy head. This parameter-sharing design lets System 1 inherit VLM pretrained knowledge while operating at high frequency via asynchronous scheduling, achieving 117.7 Hz control with a 1:4 System 2-to-System 1 frequency ratio.

Problem

Existing VLA models either run at low frequencies due to autoregressive VLM inference, or adopt dual-system designs where System 1 is a separate lightweight policy head that cannot access VLM pretrained knowledge directly — it only receives feature embeddings from System 2, limiting reasoning transfer and generalization.

Method

FiS-VLA repurposes the final few transformer blocks of a 7B LLaMA2-based Prismatic VLM as System 1, while keeping the full VLM intact for System 2. The two systems share these blocks but serve distinct roles:

• System 2 (slow, low-frequency): receives 2D RGB images and language instructions; produces intermediate latent features from an earlier LLM block via autoregressive next-token prediction.
• System 1 (fast, high-frequency): receives 3D point clouds (via a lightweight 3D tokenizer passed through the shared vision encoder), 2D images, and robot proprioceptive state; generates actions via flow-matching diffusion denoising conditioned on System 2's periodically updated latent.

Asynchronous frequency design: System 2 runs every n steps (empirically n=4 is optimal), with asynchronous sampling during training to encourage System 1 temporal consistency without System 2 re-running every step.

Dual-aware co-training: Joint loss combining diffusion denoising loss (L_fast) for System 1 and cross-entropy next-token prediction (L_slow) for System 2, preventing catastrophic forgetting of VLM reasoning. Replacing discrete action supervision with LLM-generated language plans further boosts performance to 73%.

Pretraining: 860K+ trajectories from Open X-Embodiment, DROID, ROBOMIND; fine-tuned on self-collected simulation and real-world data.

Key Contributions

• Unified dual-system VLA architecture that embeds System 1 execution within the VLM's own transformer blocks (parameter sharing), enabling System 1 to inherit VLM pretrained knowledge rather than training from scratch.
• Heterogeneous modality input design: System 2 uses 2D image + language; System 1 uses 2D image + robot state + 3D point cloud for geometry-aware, real-time control.
• Asynchronous operating frequency (1:4 ratio) with asynchronous training sampling to maintain temporal consistency across action chunks.
• Dual-aware co-training strategy (L_fast + L_slow) that preserves System 2 reasoning while enabling System 1 diffusion action generation.

Results

Simulation (RLBench, 10 tasks, Franka Panda, 100 trajectories/task): - FiS-VLA: 69% mean success rate vs. CogACT 61% (+8%) and π0 55% (+14%) - Inference speed: 21.9 Hz (action chunk=1) vs. CogACT 9.8 Hz, π0 13.8 Hz; 117.7 Hz at action chunk=8 - Superior on 8/10 tasks; largest gains on Wine at Rack (0.55 vs. 0.30 for CogACT)

Real-world (dual-arm, 8 tasks, 20 rollouts each): - Agilex robot: FiS-VLA 68% vs. π0 59% mean success rate - AlphaBot robot: FiS-VLA 74% vs. π0 61% mean success rate - Strongest gains on deformable manipulation (Fold towel: 60% vs. 40%) and spatial precision (Place Bottles at Rack: 70% vs. 55%)

Generalization (unseen objects, complex backgrounds, lighting variation): - AlphaBot unseen objects: FiS-VLA -19% drop vs. π0 -38% drop - All generalization conditions: FiS-VLA maintains >50% success across both platforms

Ablation: - Removing L_slow: 69% → 62% (−7%) - Plan-based co-training: 69% → 73% - Optimal System 1 blocks: 2 shared transformer blocks; saturates at 2 - Optimal frequency ratio: 1:4

Limitations

• System 1's shared parameter count and the System 2-to-System 1 coordination frequency are statically configured; dynamic adaptation to task complexity is not yet implemented.
• Hardware constraint prevents parallel GPU deployment of the two systems (unlike Helix), limiting achievable frequency gains from true parallelism.
• Generalization still degrades noticeably under background and lighting perturbations (−25–29% on AlphaBot, −29% on Agilex).
• Real-world evaluation only against π0; broader baseline comparison is simulation-only.

Relevance to Vision-Language Models

FiS-VLA directly addresses how VLMs can be adapted for embodied control without sacrificing their pretrained representational richness — a central challenge in VLA research. By embedding the action execution module inside the VLM's own transformer blocks rather than appending an external policy head, the work demonstrates a principled way to transfer internet-scale VLM knowledge to low-level motor control. The dual-aware co-training strategy, which jointly maintains autoregressive language/action reasoning and diffusion-based continuous action generation within a single model, is broadly relevant to anyone studying multi-task training and forgetting in foundation VLMs applied to downstream grounding tasks. The result that language plan supervision (versus discrete action tokens) for System 2 yields additional performance gains further supports the value of preserving VLM-native reasoning pipelines during robotic fine-tuning.