Skip to content

World-aware Planning Narratives Enhance Large Vision-Language Model Planner

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WAP (World-Aware Planning Narrative Enhancement) addresses the failure of LVLMs in embodied planning by augmenting training data with four cognitively structured instruction types and training via a three-stage curriculum. Applied to Qwen2.5-VL-7B and InternVL3-8B on EB-ALFRED, it yields up to +60.7 absolute success rate improvement over baseline, surpassing GPT-4o and Claude-3.5-Sonnet under closed-loop conditions.

Problem

Current LVLM-based embodied planners use environment-agnostic imitation learning that treats task instructions and environmental context as disconnected: models learn generic instruction→action mappings without grounding in visual appearance, spatial layout, object affordances, or linguistic ambiguity. This causes severe degradation on context-sensitive instructions and long-horizon tasks, and forces models to rely on privileged signals (action success feedback, task progress indicators) unavailable in real deployment.

Method

WAP constructs an augmented dataset from 16,145 ALFRED trajectories, expanding to 80,875 instruction-trajectory pairs via four instruction augmentation dimensions generated by a Qwen2.5-VL-72B teacher:

  1. Visual: adds explicit appearance descriptors (shape, color, texture) to instructions.
  2. Spatial: adds positional relationships relative to environmental landmarks.
  3. Functional: adds affordance/causal descriptions (what objects do, how they interact).
  4. Syntactic: introduces indirect references, anaphora, and narrative complexity requiring referential resolution.

Generated instructions are filtered via a semantic consistency verifier (equation 3) ensuring augmented instructions preserve original task intent. Step-wise reasoning annotations are also generated for each trajectory step using the teacher model.

Training follows a three-stage curriculum: (1) Base stage on original pairs; (2) Environmental Understanding stage adding visual+spatial augmentations; (3) Conceptual Reasoning stage adding functional+syntactic augmentations. The model operates in closed-loop with only egocentric RGB observations and complete observation history—no privileged feedback.

Key Contributions

  • Four-dimensional instruction augmentation targeting distinct cognitive capabilities (visual, spatial, functional, syntactic) grounded in cognitive development theory.
  • Demonstration that closed-loop embodied agents using only raw visual observations can match or exceed proprietary models that use privileged action feedback.
  • State-of-the-art on EB-ALFRED: Qwen2.5-VL-7B improves from 4.7 → 62.7 average SR; InternVL3-8B from 10.7 → 61.0.
  • Generalization validated on VOTA-Bench unseen split, outperforming GPT-4o across all task families.

Results

EB-ALFRED (closed-loop setting, success rate): - Qwen2.5-VL-7B: 4.7 → 62.7 (+58.0); commonsense +40 (22→62), long-horizon +68 (2→70) - InternVL3-8B: 10.7 → 61.0 (+50.3) - WAP (Qwen2.5-VL-7B) STD = 6.3 vs. Claude-3.5-Sonnet STD = 13.4 (lower = more balanced) - Outperforms GPT-4o (avg 26.0 closed-loop) and approaches Claude-3.5-Sonnet (57.3 closed-loop)

VOTA-Bench unseen split (Qwen2.5-VL-7B vs. GPT-4o): - Overall SR: WAP 64.56 vs. GPT-4o 20.36 - Overall PL: WAP 59.86 vs. GPT-4o 16.83 - WAP outperforms GPT-4o across all five task families (Examine & Light, Pick & Place, Stack & Place, Heat & Place, Cool & Place)

Ablation (Qwen2.5-VL-7B, removing components from full 62.7): - w/o Curriculum: 58.0 (−4.7); w/o Following-step Reasoning: 54.0 (−4.0); w/o First-step Reasoning: 46.7 (−7.3); w/o WAP Instruction: 47.0

Teacher model scaling: 54.67 (7B) → 57.33 (32B) → 62.67 (72B) average SR.

Limitations

  • Operates at the symbolic action level only; no continuous control parameters, requiring integration with low-level controllers for real robot deployment.
  • Validated only on household environments (ALFRED-derived); generalizability to industrial or outdoor domains with dynamic obstacles is untested.
  • Step-wise reasoning enhancement at training time does not equip the model with dynamic mid-execution error correction; recovery from execution failures remains weak.

Relevance to Vision-Language Models

This work directly advances LVLM capabilities for embodied, multi-step visuomotor tasks—a critical frontier for VLMs beyond static image understanding. The four-dimensional cognitive augmentation framework provides a principled recipe for injecting structured world-knowledge into standard VLM fine-tuning without architectural changes, generalizable to any instruction-following VLM backbone. The closed-loop, visual-only constraint is significant: it validates that sufficiently rich training narratives can substitute for privileged environmental feedback, challenging a widespread assumption in embodied AI. The curriculum learning strategy also offers a transferable training methodology for domains requiring progressive skill acquisition.