Skip to content

VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VisualPredicator introduces Neuro-Symbolic Predicates (NSPs)—Python code snippets that invoke VLMs for perceptual queries and combine them with algorithmic logic—to automatically learn abstract world models for robot planning. An online algorithm interleaves predicate invention (using VLMs), predicate selection, and goal-driven exploration to discover task-relevant abstractions from scratch, without human demonstrations. Across five simulated robotic domains, the approach achieves near-oracle solve rates while outperforming hierarchical RL and VLM-planning baselines in sample efficiency and out-of-distribution generalization.

Problem

Robot task planning requires both perceptually grounded and logically rich state abstractions, but existing approaches either use hard-coded symbolic world models (inflexible to novel environments), learn from costly human demonstrations, or restrict predicates to manually selected object features. Pure VLM planning (e.g., ViLa) fails on complex, multi-step tasks due to an inability to learn from environment feedback. There is no prior method that online-invents open-ended, visually grounded symbolic predicates without demonstration data.

Method

NSPs are Python functions with two classes: primitive NSPs that call a VLM API (with cropped images, visual object ID overlays, and previous-action context for disambiguation) to evaluate natural-language assertions about the raw state; and derived NSPs that compute truth values recursively from other NSPs (enabling logical depth, e.g., transitively checking if a block is on a plate). The online algorithm (Algorithm 1) iterates: 1. Explore: plan with current predicates/HLAs, execute, collect positive transitions, failed-skill tuples, and satisficing plans. 2. Propose NSPs: prompt a VLM with three strategies—Discrimination (contrast success/failure states for a skill), Transition Modeling (before/after snapshots), and Unconditional Generation (extend existing predicates)—to generate new predicate code. 3. Select Predicates: greedy best-first search over candidate predicate sets, scored by HLA classification accuracy plus simplicity bias (switching to a planning-efficiency objective once enough satisficing plans are found). 4. Learn HLAs: an extended cluster-and-intersect algorithm learns preconditions and effects in the learned predicate space, minimizing syntactic complexity for optimistic world models. High-level plans produced by a symbolic planner (A* with LM-Cut heuristic) are executed via the learned HLAs; failures drive further predicate invention.

Key Contributions

  • Neuro-Symbolic Predicates (NSPs): a first-order abstraction language combining VLM perceptual queries with arbitrary Python logic (conditionals, loops, recursion), supporting both primitive and derived predicates.
  • Online predicate invention algorithm: learns predicates and abstract world models purely from environment interaction, requiring no demonstration data.
  • Extended operator learner: modified cluster-and-intersect that enforces each transition is covered by exactly one HLA and minimizes syntactic complexity, improving exploration and generalization over the original.
  • Empirical evaluation against 6 methods (MAPLE, ViLa, symbolic-only predicates, ablations, oracle) across 5 PyBullet robotic domains.

Results

  • Near-oracle solve rates: VisualPredicator achieves solve rates comparable to the Oracle (manually designed predicates/operators) across all five domains (Cover, Blocks, Coffee, Cover Heavy, Balance).
  • Outperforms HRL baseline (MAPLE): MAPLE struggles even on in-distribution tasks with the same interaction budget; VisualPredicator consistently achieves higher task solve rates.
  • Outperforms VLM planning (ViLa): ViLa performs adequately on simple Cover but generates infeasible plans in complex domains (Coffee, Balance); VisualPredicator solves these reliably.
  • Outperforms symbolic predicates (Sym. pred.): Sym. pred. fails in Coffee (cannot determine jug-in-machine from object poses) and Cover Heavy (cannot identify overweight blocks); NSP perceptual grounding is essential.
  • Ablate op. underperforms: the extended operator learner reduces overfitting to early data and improves generalization, particularly in Balance and Coffee.
  • No invent baseline is substantially worse: confirming that online predicate learning provides clear benefit over the initialized underspecified abstractions.
  • Planning efficiency: VisualPredicator uses planning budget close to the theoretical minimum (1 plan per task) in most domains after learning converges.

Limitations

  • Evaluated only in simulation (PyBullet); real-world transfer, partial observability at scale, and sim-to-real gap are unaddressed.
  • Assumes predefined motor skills (pick, place, etc.) and an object-centric state representation provided externally.
  • VLM queries are imperfect (acknowledged; mitigated via context/ID overlays, but not eliminated), and VLM-generated predicate code can have low precision, requiring selection/filtering.
  • Predicate selection via GBFS is feasible for the current search spaces but may not scale to much larger environments without local hill climbing substitution.
  • No recovery mechanism for failed plans during execution; failures are used for learning but not corrected mid-plan.
  • Relies on knowing when a skill has successfully executed (closed-loop skill feedback assumed).

Relevance to Vision-Language Models

This paper demonstrates a concrete, task-grounded use of VLMs as programmatic perception oracles within robot planning: VLMs are called inside generated Python predicates to evaluate natural-language assertions about cropped image regions, rather than being used end-to-end for planning. This is directly relevant to VLM research because it exposes both the capability (VLMs can ground abstract concepts like "jug inside machine" reliably when given the right context/visual markup) and the limitation (VLM planning alone, as in ViLa, fails on complex multi-step tasks without a learned world model). The Set-of-Mark visual grounding technique (overlaying numeric IDs on objects) is adopted to improve VLM reference resolution, echoing findings from GPT-4V grounding work. For VLM researchers, VisualPredicator offers a structured neuro-symbolic architecture that systematically addresses VLM unreliability—temporal context, previous-action conditioning, object disambiguation—pointing toward a productive hybrid paradigm for grounded reasoning.