Do Coding Agents Need Executable World Models, Simplification, and Verification to Solve ARC-AGI-3?

🕒 Published (v1): 2026-07-16 20:18 UTC · Source: Arxiv · link

Why this paper was selected

Ablates ARC-AGI-3 agent components (world model, simplification, verification); direct scaffold design signal

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper runs a controlled nested ablation study on four coding agents for ARC-AGI-3, isolating the contribution of three design choices: maintaining an executable world model, applying scheduled simplification, and requiring exact observation-replay verification. The headline result is that model capability and reasoning effort dominate component effects, but the full verification treatment consistently ranks first and saturates the public game set with gpt-5.6-sol (~99% RHAE).

Problem

Prior ARC-AGI-3 work bundled executable world modeling, scheduled simplification, and replay verification into a single proof-of-concept agent, leaving the attribution of performance gains to individual components unresolved. The paper addresses this causal question: when the underlying coding agent is already capable, does each component independently contribute?

Method

Four nested agent variants are defined, each adding one component over the previous:

1. twma_v1.2 — textual world model only (no persistent executable simulator required, but REPL/code execution still allowed)
2. ewma_v1.2 — adds a requirement to maintain executable world-model and planning code with flexible interfaces
3. ewma_s_v1.2 — adds controller-triggered simplification/generalization passes at scheduled points (light prompt on level 1, four increasingly broad passes on later states)
4. ewma_sv_v1.2 — adds a fixed four-part interface contract (transition engine, initial-state reconstruction, renderer, planner), exact observation-replay verification against all recorded attempts, and planner verification on completed levels

All variants share the same thin prompt-level controller wrapping a Codex CLI (v0.128.0) coding-agent runtime. The controller is a small state machine (normal/reset/stuck protocols) and does not contain game-solving logic. Agents run in isolated Docker environments with no internet access and no game-name cues. The primary metric is RHAE (Relative Human Action Efficiency), rewarding level completion penalized by action count relative to a human baseline. The 4×2×2 design (4 variants × {gpt-5.4, gpt-5.5} × {high, xhigh} reasoning effort) covers all 25 public ARC-AGI-3 games, each played once per condition. Follow-up runs test textual and verification variants with gpt-5.6-sol at xhigh and max effort.

Cost tokens are reported as a normalized proxy: \(C = T_\text{cached}/60 + (T_\text{input} - T_\text{cached})/6 + T_\text{output}\), weighting by relative API prices.

Key Contributions

• Controlled nested ablation separating executable world modeling, simplification, and verification within a single coding-agent family
• Demonstration that requiring a persistent executable deliverable is not universally beneficial: textual variant outperforms the flexible-interface executable variant in both gpt-5.5 settings
• Simplification improves performance in 3 of 4 model–effort settings (the weakest setting is the sole exception)
• Verification treatment ranks first in all 4 main-study settings, but uses substantially more resources
• Hardened evaluation harness (Docker isolation, no game-name leakage, restricted proxy) that closes concrete information-leakage channels identified in prior audits
• gpt-5.6-sol saturation result: verification variant solves all 25 public games at both reasoning efforts with ~99% RHAE and fewer than half the human baseline action count

Results

• Dominant effect: Every agent variant improves with stronger model and higher reasoning effort; cross-variant differences within a fixed model–effort setting are smaller than anticipated
• Executable vs. textual: Flexible-interface executable variant underperforms textual variant in both gpt-5.5 settings; maintaining an executable simulator is not universally helpful
• Simplification: Improves performance in 3/4 model–effort settings; no improvement only in the weakest (gpt-5.4, high) setting
• Verification: Ranks first in all 4 main-study (gpt-5.4/gpt-5.5) conditions, at the cost of substantially higher resource use
• gpt-5.6-sol follow-up: Verification variant achieves ~99% RHAE on all 25 public games at both xhigh and max reasoning effort, using <50% of human baseline actions; interpreted as public-set saturation only (held-out performance untested; model postdates the public games)

Limitations

• Each condition evaluated with a single playthrough per game (no repeated trials), so variance across runs is unmeasured
• Nested ablation ladder does not test all component combinations (e.g., verification without simplification is not evaluated)
• Final comparison (ewma_s vs. ewma_sv) changes both the verification objective and the supporting workspace, so it estimates the complete verification treatment effect, not isolated verification
• gpt-5.6-sol results apply only to the 25 public games; held-out (semi-private/fully private) performance is untested and the model postdates the public set, raising contamination concerns
• Prompt-cache hit rates in subscription-backed Codex runs differ substantially from API-key runs, complicating cost comparisons
• Hardened harness does not formally prove security against an adversarial agent; residual contamination (game recognition from observations alone) is acknowledged but not measured

Relevance to Agentic AI / LLM Agents

This paper directly addresses a core design question for agents operating in novel interactive environments: whether externalizing the environment hypothesis as executable code provides a systematic advantage over maintaining it implicitly in the model's context. The finding that textual agents can outperform flexible-interface executable agents challenges the default assumption that programmatic world models always improve grounding. The verification-via-replay pattern—using the agent's own interaction history as a partial verifier for its world model—is a principled, deployment-ready instantiation of the generate-and-verify paradigm applicable to any agentic setting where ground-truth verifiers are unavailable. The ablation methodology itself is a contribution to rigor in agentic benchmarking, where most published comparisons conflate model, harness, and component effects.