Skip to content

APIVOT: Adaptive Planning with Interleaved Vision-Language Thoughts

🕒 Published (v1): 2026-07-09 01:02 UTC · Source: Arxiv · link

Why this paper was selected

Stanford (Haber lab); interleaved vision-language thoughts for long-horizon robot planning — actionable VLA scaffold

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

APIVOT is a VLM-based planner that adaptively interleaves language tokens (for semantic reasoning) with visual thought tokens (as imagined future states for geometric feasibility verification) during long-horizon robot planning. It is trained via a three-stage curriculum on a kitchen manipulation benchmark and outperforms general-purpose VLMs and planning frameworks while reducing reasoning token usage through selective visual thought generation.

Problem

Existing LLM/VLM planners produce semantically plausible but geometrically infeasible plans, failing when success depends on spatial constraints (object collisions, limited free space, arrangement feasibility). Prior geometric grounding approaches couple planners with external verifiers or motion planners, leaving geometry outside the model's internal reasoning process. There is no principled mechanism for deciding when to reason visually versus linguistically.

Method

APIVOT generates a multimodal reasoning trace \(R = r_1 \oplus \cdots \oplus r_m\), where each step \(r_j\) contains text tokens \(t_j\) and optionally a fixed-length span of \(K\) visual tokens \(v_j\) (a "visual thought"). The decoder hidden states \(H_j \in \mathbb{R}^{K \times d}\) at visual token positions encode a latent future scene representation that conditions all subsequent autoregressive decoding.

Visual thoughts are aligned to ground-truth future RGB observations via a cosine similarity loss on projected latent states: $\(\mathcal{L}_\text{vis} = \frac{1}{|B_\text{vis}|} \sum_{j \in B_\text{vis}} 1 - \cos\!\left(f_\phi(H_j),\, \text{AvgPool}\bigl(E(I_j^\text{gt})\bigr)\right)\)$

The full training objective is \(\mathcal{L}_\text{total} = \mathcal{L}_\text{CE,plan} + \lambda_\text{CE,trace}\mathcal{L}_\text{CE,trace} + \lambda_\text{vis}\mathcal{L}_\text{vis}\).

Training proceeds in three stages: 1. Stage 1 (comprehension): Ground-truth visual features are injected at all visual token positions; model learns to use visual context for planning (no generation, no visual loss). 2. Stage 2 (generation): Model autoregressively generates its own latent visual states and is supervised with \(\mathcal{L}_\text{vis}\) to align them to ground-truth future images. 3. Stage 3 (adaptive selection): Training traces include visual thoughts only for subgoals with geometric constraints (identified by simulator-derived heuristics on free-space margins and collision sensitivity); model learns when to emit visual thoughts.

Reference traces are constructed by decomposing demonstrations into subgoals, classifying each as symbolic-precondition, current geometric, or future geometric feasibility, and expanding structured skeletons via a high-capacity LM. The base model is Qwen3-VL-8B-Instruct fine-tuned with LoRA.

Key Contributions

  • APIVOT: a VLM planner that generates visual thoughts as imagined future states within its reasoning trace for internal geometric verification during long-horizon planning.
  • A three-stage SFT curriculum (comprehension → generation → adaptive selection) that progressively teaches the model to reason over, produce, and selectively invoke visual thoughts.
  • A reference trace construction pipeline that derives modality labels from simulator geometric metrics (free-space margins, object footprints, collision tightness) and expands them into natural-language traces.
  • Demonstration that adaptive visual thought generation improves both task success and reasoning efficiency simultaneously.

Results

  • APIVOT achieves a mean task success rate of 0.419, an 8.1 percentage point improvement over the top-performing baseline across all task families (CONTAINMENT, SORTING, STORING LEFTOVERS).
  • In spatially constrained settings, the performance gap widens from 7 to 17 points as geometric complexity increases.
  • Adaptive APIVOT retains 91% of the performance of the visual-thought-at-every-step upper bound while reducing token usage by 39%.
  • APIVOT outperforms all baselines: Gemini-3.1-Pro (with extended thinking), Gemini-ER-1.5, Qwen3-VL-8B-Instruct, Qwen3-VL-8B-Thinking, VLM-TAMP, Reflect-VLM, and FastDownward.
  • APIVOT consistently outperforms language-only VLMs across varying reasoning budgets, indicating visual thoughts are a more compact representation than equivalent language descriptions.

Limitations

  • Evaluated only in simulation (KitchenWorlds); real-world transfer is unvalidated.
  • Perception (object detection, localization) is assumed as input; the planner is not evaluated end-to-end with a perception module.
  • Placement actions use image-space coordinates lifted to world-frame via depth and known camera extrinsics — requires calibrated sensing.
  • Action space is restricted to three parameterized skills (open, pick, place); generalization to richer skill sets is unvalidated.
  • Reference trace construction relies on a high-capacity LM to expand subgoal skeletons, introducing a data pipeline dependency on that model's quality.
  • Training data is limited to 2,000 examples per task family; scalability is not assessed.

Relevance to Vision-Language Models

APIVOT directly advances VLM capabilities by demonstrating how a VLM can generate and reason over self-produced visual representations rather than only consuming external images, a step toward VLMs that internally simulate the world. The adaptive modality selection mechanism — choosing language versus visual tokens based on task demands — is a concrete instantiation of efficient multimodal chain-of-thought, addressing a key open question in VLM reasoning about when visual computation is worth its cost. The three-stage curriculum for visual thought grounding (comprehension → generation → selection) provides a transferable training recipe for injecting spatial reasoning into any pretrained VLM. This work connects to broader efforts in multimodal reasoning, visual chain-of-thought, and vision-language-action models, offering a planning-centric perspective that complements perception-focused VLM work.