Skip to content

Simulation to Rules: A Dual-VLM Framework for Formal Visual Planning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Dual-VLM bridges formal PDDL planning with visual scene understanding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMFP is a dual-VLM framework that autonomously generates both PDDL problem and domain files from visual inputs for formal long-horizon planning — the first system to do so without human feedback or environment access. A fine-tuned SimVLM (Qwen2-VL-7B) simulates action consequences, while a large GenVLM (GPT-4o) generates and iteratively refines PDDL files by comparing symbolic execution against simulated outcomes. Evaluated on six grid-world domains and complex 3D tasks, VLMFP substantially outperforms baselines that rely on pre-defined domain files.

Problem

VLMs lack precise spatial reasoning and long-horizon planning capability. PDDL planners solve long-horizon problems exactly but require expert-crafted domain files and cannot parse visual inputs. Prior VLM+PDDL works can extract PDDL problem files from images but still assume access to ground-truth PDDL domain files (encoding action dynamics and constraints), leaving domain-file generation unsolved without human expertise or direct environment interaction.

Method

VLMFP decomposes the visual-to-PDDL pipeline across two specialized models:

SimVLM (fine-tuned Qwen2-VL-7B on 430k grid-world simulation datapoints): given domain description \(n_d\), problem image \(i_p\), and action sequence \(\pi\), it (1) describes spatial layout in natural language \(n_p\), (2) simulates step-by-step action consequences, and (3) determines whether the goal is reached.

GenVLM (GPT-4o): given \((n_d, i_p, n_p)\), generates initial PDDL domain \(f_d^{(0)}\) and problem \(f_p^{(0)}\) files, then iteratively refines them.

The iterative loop has four stages: 1. Candidate generation: \(n_p = V_S(n_d, i_p)\); \((f_d^{(0)}, f_p^{(0)}) = V_G(n_d, i_p, n_p)\). 2. Prescreening: syntactic and semantic validity check (max 5 regeneration rounds). 3. Simulation consistency checking: bidirectional Exploration Walk (EW) score

\[\hat{m}_\text{EW}(\hat{d},\hat{p}) = 2\left(\frac{1}{T_\text{max}}\sum_{T=1}^{T_\text{max}}\mathbb{E}_{q \sim P_\text{sim,T}}[E_{f_d,f_p}(q)]^{-1} + \frac{1}{T_\text{max}}\sum_{T=1}^{T_\text{max}}\mathbb{E}_{q \sim P_{f_d,f_p,T}}[E_\text{sim}(q)]^{-1}\right)\]

measures alignment between SimVLM execution and PDDL execution; discrepancies are summarized as natural-language feedback \(s\). 4. PDDL updating: \(V_G\) refines \((f_d^{(t-1)}, f_p^{(t-1)})\) using feedback \(s\).

The domain file is reusable across all instances; problem files are adapted efficiently via in-context examples.

Key Contributions

  • 430k-datapoint simulation dataset spanning 6 grid-world domains with varying map sizes, obstacle probabilities, and 5–6 distinct visual appearances per domain, used to fine-tune Qwen2-VL-7B as SimVLM.
  • VLMFP: first framework generating both PDDL domain and problem files from raw visual input without human feedback or environment access.
  • Bidirectional EW score as a model-free fidelity metric that produces natural-language correction signals for iterative refinement.
  • Demonstration of multi-level generalization: unseen instances, unseen visual appearances, and altered game rules.
  • Scalability to complex 3D long-horizon tasks (multi-robot collaboration, assembly) with partial observability.

Results

  • SimVLM accuracy (string matching): average 87.3% seen / 86.0% unseen appearances across 6 domains and 4 output types; average seen-to-unseen drop of only 1.3%.
  • Grid-world planning success (unseen instances): 70.0% on seen appearances, 54.1% on unseen appearances with GPT-4o as GenVLM.
  • Baseline comparison: outperforms best baseline CodePDDL\(_\text{GPT-4o}\) by +39.3% (seen appearances) and +21.8% (unseen appearances).
  • 3D tasks (MultiRob, Assembly): 86.4% success on unseen instances in seen appearances, 79.8% in unseen appearances.

Limitations

  • SimVLM fine-tuned exclusively on six specific grid-world domains; recognition failures occur for novel-appearance objects not present in training (e.g., Sokoban unseen Task Description drops 23.3% vs. seen).
  • GenVLM relies on GPT-4o via API — closed, costly, and subject to latency; no open-source GenVLM alternative evaluated.
  • Prescreening caps at 5 regeneration rounds; failure to pass prescreening terminates that branch without recovery.
  • Evaluation limited to grid worlds and two 3D scenarios; transfer to unstructured real-world visual environments (non-discrete action spaces, continuous state) is undemonstrated.
  • Paper text is truncated; explicit limitations section (if present) may contain additional caveats not captured here.

Relevance to Vision-Language Models

This work directly targets a core VLM deficiency — weak spatial and long-horizon reasoning — by training a small VLM (7B) as a grounded simulation oracle rather than as a direct planner, sidestepping the need for VLMs to master PDDL syntax while still contributing perceptual intelligence. The fine-tuning strategy on 430k structured simulation traces with spatial grounding is a concrete recipe for improving spatial reasoning in small VLMs. The dual-VLM division of labor (specialized perception model + large general reasoning model) is a reusable architectural pattern for any VLM application requiring both visual grounding and structured symbolic output. For the VLM+planning literature, VLMFP closes the last open loop — autonomous domain-file generation from vision — making fully unsupervised visual-to-formal-plan pipelines practical.