Skip to content

ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ThinkAct is a dual-system VLA framework that uses reinforcement learning to train a multimodal LLM to generate embodied reasoning plans, which are then compressed into a visual plan latent that conditions a downstream diffusion-based action model. The key novelty is an action-aligned visual reward (goal completion + trajectory consistency) that grounds reasoning in physical plausibility without requiring expensive annotated CoT traces.

Problem

Existing VLA models map vision-language inputs directly to actions end-to-end, bypassing explicit planning, which limits performance on long-horizon tasks, out-of-distribution scenarios, and adaptive replanning after failures. Prior CoT-augmented VLAs either depend on costly human-annotated reasoning traces (SFT-only) or use QA-style RL rewards that cannot bridge reasoning with real action execution.

Method

ThinkAct uses a two-module architecture:

Reasoning MLLM (Fθ, Qwen2.5-VL 7B): Given observation o_t and instruction l, the MLLM autoregressively generates (1) textual reasoning tokens and (2) a visual plan latent c_t that encodes a predicted 2D gripper trajectory τ = [p_k]^K as text-formatted point sequences.

Action-aligned reward for GRPO fine-tuning: Two visual rewards supervise the MLLM: - Goal reward r_goal: L2 proximity between predicted start/end gripper positions and ground-truth positions from an off-the-shelf trajectory detector. - Trajectory reward r_traj: Dynamic Time Warping (DTW) distance between predicted and demonstrated full trajectories. - Combined: r = 0.9(0.5·r_goal + 0.5·r_traj) + 0.1·r_format.

GRPO optimizes the MLLM using group-relative advantage estimation over M=5 rollouts.

DiT-based Action Model (πϕ, 432M params): Pre-trained on Open X-Embodiment. At adaptation, c_t is projected via a Q-Former (32 queries) into the action model's input space; only the state encoder, projector, and action model weights are updated via imitation learning. The two modules run asynchronously: the MLLM reasons every N steps while the action model executes at every step.

Training stages: (1) SFT cold-start on OXE trajectories + robotic QA; (2) GRPO RL fine-tuning with action-aligned rewards; (3) reasoning-enhanced action adaptation with Fθ frozen.

Key Contributions

  • Dual-system VLA framework connecting slow MLLM reasoning to fast DiT-based action execution via a compact visual plan latent.
  • Action-aligned visual reward combining goal-completion (endpoint L2) and trajectory-alignment (DTW) signals, enabling RL fine-tuning without environment-specific simulators.
  • Asynchronous inference design allowing the reasoning and action modules to operate at different cadences.
  • Demonstrated emergent self-correction: by extending MLLM input to a short video segment, the model detects execution failures and replans without task-specific supervision.

Results

Robot manipulation (SimplerEnv, task success rate): - ThinkAct achieves 71.5% (Google-VM), 65.1% (Google-VA), 43.8% (Bridge-VM) overall — best among all methods including Magma (68.4%, 62.6%, 35.4%) and DiT-Policy (56.0%, 48.2%, 32.4%). - Improvements over baseline DiT-Policy: +15.5% (Google-VM), +16.9% (Google-VA), +11.4% (Bridge-VM).

LIBERO long-horizon manipulation: - ThinkAct: 84.4% overall (best), vs. CoT-VLA 83.9% and DiT-Policy 76.8%. - LIBERO-Long (hardest subtask): 70.9% vs. CoT-VLA 69.0%, OpenVLA 53.7%.

Embodied reasoning benchmarks: - EgoPlan-Bench2: 48.2% overall — best, +2.5% over second-best Magma (45.7%). - RoboVQA: 59.8 BLEU overall — best, +4.1 over second-best Magma (55.7%). - OpenEQA: 56.2% overall — best among non-GPT-4V models.

Few-shot adaptation (10-shot LIBERO): - Outperforms Magma by 9.5% on LIBERO-Spatial, 7.3% on LIBERO-Goal.

Ablation: Removing r_traj or r_goal both degrade performance; removing both reduces to near-SFT baseline, confirming the criticality of action-aligned visual rewards over QA-only RL.

Inference overhead: 17% longer than OpenVLA on LIBERO, yielding +2.8–15.3% gains across task categories.

Limitations

  • Inherits hallucination tendencies from the pretrained MLLM base (Qwen2.5-VL), which can produce plans referencing incorrect object attributes or spatial relationships.
  • Reward signals use only 2D gripper traces; contact-rich or 3D dynamics are not captured in the current reward formulation.
  • Self-correction requires extending the MLLM input to a video segment, adding latency and not yet evaluated quantitatively.
  • Evaluation is limited to simulation (SimplerEnv, LIBERO); real-world robot deployment is not demonstrated.

Relevance to Vision-Language Models

ThinkAct directly extends VLMs into the embodied action domain by applying R1-style GRPO reinforcement learning with visually grounded, action-specific rewards rather than QA accuracy — a significant methodological contribution for VLM researchers studying RL-based reasoning. The visual plan latent mechanism offers a general pattern for bridging VLM "thinking" tokens to downstream specialized models without end-to-end retraining, relevant to work on VLM chain-of-thought, tool use, and modular architectures. The result that RL with action-aligned rewards outperforms SFT-on-CoT with far less annotation cost is directly relevant to the debate on how to scale VLM reasoning for grounded tasks.