Skip to content

Agent2World: Learning to Generate Symbolic World Models via Adaptive Multi-Agent Feedback

🕒 Published (v1): 2025-12-26 18:54 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agent2World is a three-stage multi-agent framework (Deep Researcher → Model Developer → Testing Team) that generates symbolic world models (PDDL domains, executable simulators) from natural language descriptions. Beyond inference-time use, the developer–tester interaction loop doubles as a data engine: multi-turn repair trajectories filtered by verifier-guided rejection sampling are used to fine-tune a smaller LLM, yielding an average 30.95% relative gain over the base model.

Problem

Training LLMs to synthesize symbolic world models is hampered by (1) scarce large-scale verifiable supervision and (2) static validation pipelines (parsers, fixed test suites) that catch syntactic errors but miss behavior-level failures—unreachable goals, inconsistent state updates, missing preconditions—that only appear under interactive execution.

Method

The framework operates in three sequential stages:

  1. Knowledge Synthesis — A Deep Researcher agent (ReAct loop + Serper web search) fills specification gaps and produces a structured intermediate representation.
  2. World Model Generation — A Model Developer agent (ReAct loop + sandboxed code execution) iteratively implements and self-debugs the world model (PDDL or Python).
  3. Evaluation-Driven Refinement — A two-agent Testing Team conducts (a) Unit Testing: auto-generates Pytest-style unit tests targeting predicates, actions, and invariants; (b) Simulation Testing: interacts with the environment in a ReAct loop, collecting trajectories to surface execution-time failures. Diagnostic reports are fed back to the Model Developer, triggering additional repair rounds (2–3 iterations depending on benchmark complexity).

Training pipeline: The developer–tester loop is formalized as an MDP. Multi-turn interaction trajectories are generated with GPT-4.1-mini; verifier-guided rejection sampling retains only trajectories whose final world model passes sandbox execution and all Testing Team checks. The resulting 1,526 verified trajectories (spanning PDDL, MuJoCo-style, text games, MCP-style) are used for SFT on Llama-3.1-8b (5 epochs, lr=1e-6, max 30k tokens).

Key Contributions

  • Multi-agent specialization with adaptive, behavior-aware test synthesis replacing static validators.
  • Dual use of the inference-time loop as a labeled-data-free training data engine via rejection sampling.
  • State-of-the-art results across three heterogeneous symbolic world-model benchmarks (PDDL, code-based, text games).
  • Empirical demonstration that surface-level accuracy (structural similarity) is a poor proxy for behavioral correctness, motivating execution-grounded evaluation.
  • 1,526-trajectory verified SFT dataset covering four world-model representation types.

Results

Text2World (PDDL, 103 domains) — GPT-4.1-mini backbone: - Agent2WorldMulti: Executability 93.1% vs. Text2WorldEC=3 baseline 78.2% (+14.9 pp); F1AVG 75.4 vs. 60.1 (+15.3 pp) - Llama SFT: Executability +16.9 pp over Agent2WorldMulti; F1AVG +6.4 pp

CWMB (18 MuJoCo-style environments) — GPT-4.1-mini: - Agent2WorldMulti overall normalized return R = 0.4811 vs. GIF-MCTS best of 0.3752 (+0.1059) - Llama SFT: overall accuracy 0.6857 vs. prior best 0.5983; R 0.3197 vs. 0.2296

ByteSized32 (32 text games) — GPT-4.1-mini: - Agent2WorldMulti: Physical Alignment 0.4768 vs. Self-Consistency 0.3562 (+0.1206); Runnable Game 0.8958 vs. 0.7708 - Llama SFT: Specification Compliance (Critical Actions) +0.2944; Winnable Game +0.0388

Pairwise win rates vs. prior SOTA: ByteSized32 87% wins, CWMB 66.7% wins, Text2World 31.2% wins.

Ablation (CWMB discrete): Removing Unit Tester drops accuracy by 0.3008 and R by 0.4470; removing Deep Researcher drops R by 0.3926.

Limitations

  • Higher token cost than single-agent baselines due to proactive testing loop; exact overhead reported only in appendix.
  • SFT evaluated only on Llama-3.1-8b; generalization to other open-weight models unverified.
  • Refinement iteration count is hand-tuned per benchmark (2 for Text2World/ByteSized32, 3 for CWMB), not automatically determined.
  • Web search is used during both training data generation and inference; denylist mitigates but does not fully eliminate leakage risk.
  • ByteSized32 physical alignment and winnability scores remain low in absolute terms even after SFT (0.1040 and 0.1042), indicating the task is far from solved.
  • Training trajectories generated exclusively with GPT-4.1-mini; potential bias if the same model is used for evaluation.

Relevance to Harnesses / Meta-Harnesses

Agent2World is a canonical harness-as-data-engine pattern: the multi-agent inference harness (researcher → developer → tester) is not just an execution scaffold but also the generator of its own training corpus via rejection sampling, with no human annotation required. This closes the loop between meta-harness orchestration and model improvement, directly relevant to researchers designing self-improving or self-supervised harnesses. The adaptive testing component—where the Testing Team synthesizes new checks conditioned on observed failures rather than running a fixed suite—instantiates a meta-level quality loop that could be transplanted into other harness architectures. The MDP formalization of the developer–tester interaction also provides a principled template for framing multi-agent harness loops as learnable policies.