Skip to content

Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction

🕒 Published (v1): 2026-07-09 08:29 UTC · Source: Arxiv · link

Why this paper was selected

ZendoWorld: active visual concept induction benchmark — tests hypothesis-forming and experiment-designing agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ZendoWorld is a controlled interactive benchmark where agents must perceive 3D visual scenes, induce hidden logical rules, and actively design informative experiments to test hypotheses — the first benchmark combining all three capabilities in a visually grounded setting. Evaluation of VLM, Bayesian, neuro-symbolic, and program synthesis agents reveals that none reliably solve the full loop, with VLMs achieving only 44.5% win rate versus 73.3% for humans. Key findings: labeling accuracy is dissociated from rule recovery, perception and induction are independent bottlenecks, and VLM-generated experiments are near-uninformative.

Problem

Existing visual induction benchmarks (ARC-AGI, Bongard, RAVEN) treat rule discovery as a passive, one-shot task over symbolic or grid inputs, omitting the iterative hypothesis-test-revise cycle central to scientific reasoning. Benchmarks that do include active experimentation (AutumnBench, ARC-AGI-3) still rely on symbolic rather than visual input. No prior benchmark jointly requires 3D visual perception, explicit hypothesis formulation, and active experiment design within a single interaction loop.

Method

ZendoWorld is an episodic multi-turn game derived from the inductive logic game Zendo. Each episode is defined by a latent rule \(r^* : S \to \{0,1\}\) drawn from a Prolog-based DSL grammar with predicates over object attributes (color, shape, orientation), spatial relations, and quantifiers. The agent: 1. Receives two labeled seed scenes \((x_\text{pos}, 1), (x_\text{neg}, 0)\). 2. Proposes new scenes \(e_t \in S\) with predicted label \(\hat{y}_t\); receives ground-truth feedback \(y_t = r^*(e_t)\). 3. May commit a rule hypothesis \(h_t\) after a correct label prediction; if wrong, receives a counterexample. 4. Terminates on correct rule equivalence or after \(|D| \leq 30\) examples.

Scenes are 3D renders of 1–7 colored geometric objects (blocks, wedges, pyramids) generated via a Blender–Prolog pipeline over \(\approx 10^{18}\) configurations. Rules are partitioned into three complexity tiers: First-Order (counting single attributes), Second-Order (parity/comparative quantifiers), and Compositional (spatial/logical connectives). Hypothesis quality is measured via structural F1 over the rule parse tree. Experiment quality is measured by expected information gain (EIG) relative to the hypothesis posterior.

Four agents are evaluated: an Oracle (program synthesis + pretrained perception + DSL + heuristic experiment selection), a VLM Agent (end-to-end GPT-5-mini, no DSL), a Bayesian Agent (LLM + Sequential Monte Carlo particle filtering over hypotheses), and a VLP Agent (VLM-based concept extraction + general-DSL program synthesis). A human study with 19 participants provides a reference baseline.

Key Contributions

  • ZendoWorld benchmark: 22 games spanning first-order, second-order, and compositional rules plus one OOD rule, with controlled 3D visual input and precise DSL-based ground-truth feedback.
  • Evaluation protocol decomposing perception, induction, and experimentation as independently measurable failure modes.
  • Empirical demonstration that label accuracy and rule recovery are dissociated (Bayesian Agent: 75.2% label accuracy, 13.6% win rate).
  • Human behavioral dataset (19 participants, 10 data points per game) establishing human-agent gaps per rule complexity class.
  • EIG analysis showing VLM-generated experiments are near-uninformative relative to random baselines.

Results

  • End-to-end win rate: Oracle 95.5%, Human 73.3%, VLM 44.5%, VLP 18.2%, Bayesian 13.6%.
  • Label accuracy during interaction: Bayesian 75.2% (highest), Oracle 68.2%, VLP 66.6%, VLM 56.7%, Human 53.9% — highest label accuracy does not correlate with win rate.
  • Effect of rule complexity (VLM Agent): 67.5% on first-order, ~30% on second-order and complex; Bayesian fails entirely on second-order and complex games.
  • OOD task: 7/10 human participants solved it; no agent succeeded on visual input; only VLP solved it in the symbolic-input ablation.
  • Symbolic-input ablation (replacing rendered scenes with Prolog descriptions): VLM +29.4% (57.6%), Bayesian +77.9% (24.2%), VLP +166.5% (48.5%) — confirming perception as a major bottleneck.
  • Experimentation quality: VLM agents generate experiments with near-zero EIG; Oracle heuristic experimentation outperforms Oracle with random experiments, confirming experimentation strategy matters.
  • Human convergence: Humans produce more stable structural F1 trajectories and converge faster on complex rules than VLM agents; both humans and VLMs show late-stage F1 drops before committing incorrect rules.

Limitations

  • Rule space is finite and DSL-defined; the OOD task is the only probe of out-of-distribution generalization.
  • Oracle depends on pretrained perception modules and full DSL access, setting an upper bound that is not achievable by general agents without environment-specific priors.
  • Human study is small (19 participants, subset of 22 games) and browser-based, limiting statistical power.
  • Equivalence checking via LLM translation for natural-language hypotheses introduces noise (invocation rates reported per agent in appendix).
  • Evaluation uses a single VLM backbone (gpt-5-mini-2025-08-07) for fair comparison; backbone ablations are secondary results.
  • Scene generation is limited to simple geometric objects; transfer to naturalistic visual domains is not studied.

Relevance to Agentic AI / LLM Agents

ZendoWorld directly probes a capability gap central to autonomous AI agents: the ability to close the perception–hypothesis–experiment loop rather than treating reasoning as a one-shot inference problem. The finding that VLM agents propose near-uninformative experiments — despite strong few-shot reasoning on static benchmarks — points to a fundamental limitation in current LLM-based agentic pipelines: they lack principled uncertainty-driven exploration strategies. The dissociation between label accuracy and rule recovery is a concrete warning for agent evaluation methodology, showing that standard classification metrics can mask total failure at the underlying task. For researchers building agentic systems for scientific discovery or autonomous experimentation, ZendoWorld provides a controlled, reproducible testbed to measure progress on active hypothesis revision.