Skip to content

Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1

🕒 Published (v1): 2026-07-07 19:49 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces two agentic harnesses — an Explorer-Definer Pipeline and a Reflective Orchestrator — that solve ARC-AGI-1 tasks under a strict budget using an open-weight model with no benchmark-specific fine-tuning. The pipeline reaches 57.50% pass@2 at $0.25/task; the orchestrator reaches 67.25% pass@2 at $0.62/task, a combined +52 pp lift over a one-shot baseline. A generation-vs-selection diagnostic ties the orchestrator's design to a falsifiable architectural prediction, which it empirically confirms.

Problem

The sub-$1/task region of the ARC-AGI-1 cost–accuracy frontier is largely unoccupied: strong results require either heavy test-time compute (evolutionary search, exhaustive sampling) or benchmark-specific training/fine-tuning. The gap between cheap one-shot LLM calls (~16%) and competitive systems (≥50%) is not fully explained by model capability — it remains unclear how much is recoverable through harness architecture alone, holding model and training regime constant.

Method

All architectures use DeepSeek V3.2 in non-thinking mode (no ARC-specific training). Four architectures of increasing structural complexity are compared on the same 400-task public evaluation set:

  1. One-shot baseline: single LLM call, structured JSON output.
  2. CoT baseline: adds a reasoning field to the JSON schema, structurally forcing externalized reasoning before grid generation.
  3. Explorer-Definer Pipeline: two-stage agentic decomposition.
  4. Phase 1: \(N\) parallel PatternExplorer agents each equipped with think(thought) (private scratchpad) and note_pattern(pattern) tools; they analyze training pairs independently and emit free-form natural-language pattern descriptions.
  5. Phase 2: a single TransformationDefiner agent reads all \(N\) explorer findings, synthesizes a Python transformation function, and validates it via deterministic execution against training pairs. Best-of-\(M\) sampling is used; candidates are deduplicated by test-prediction before selecting top-2 for pass@2.
  6. Reflective Orchestrator: extends the definer with a mid-loop spawn_explorers tool. When the current transformation hypothesis fails particularly badly on training pairs, the definer branches out of its synthesis frame and spawns \(K=2\) fresh focused exploration agents conditioned on a structured failure summary, then resumes synthesis. The loop-exit criterion is model-mediated.

Canonical configuration: \(N=5\), \(t=0.5\) (explorers), definer temperature 0, \(M=5\). A Pareto sweep over \(N \in \{1,2,5\} \times t \in \{0.0,0.5,1.0\} \times M \in \{1,\dots,5\}\) (45 configurations from 18 runs) maps the cost–accuracy frontier. The unbiased pass@\(k\) estimator

\[\widehat{\text{pass@}k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\]

is used alongside naive pass@\(k\) to separate generation capacity from selection quality.

Key Contributions

  • Explorer-Definer Pipeline: 57.50% pass@2 at $0.25/task (+42 pp over one-shot, +27.5 pp over CoT) via explicit decomposition of pattern discovery and program synthesis.
  • Reflective Orchestrator: 67.25% pass@2 at $0.62/task (+9.75 pp over pipeline, 95% CI [+6.50, +13.25]) via mid-loop adaptive re-exploration when synthesis is failing.
  • Generation-bound diagnostic: unbiased pass@\(k\) analysis shows train-score selection captures ~95% of the candidate ceiling, establishing the pipeline is generation-bound not selection-bound — motivating the orchestrator as an architectural test of that prediction.
  • Pareto sweep: identifies explorer count \(N\) as the dominant cost–accuracy lever; best-of-\(M\) definer sampling appears to saturate; intermediate temperature often outperforms extremes.
  • Ablations: removing the definer's think scratchpad costs −5.75 pp pass@2; disabling train-feedback refinement costs only −0.75 pp.
  • Cross-model validation: matched 99-task evaluation on Qwen3-235B follows the same lift trajectory, suggesting the harness architecture — not the specific model — drives the gain.

Results

  • One-shot baseline: 15.50% pass@2
  • CoT baseline: 30.00% pass@2
  • Explorer-Definer Pipeline: 57.50% pass@2 at $0.25/task (+42.00 pp over one-shot)
  • Reflective Orchestrator: 67.25% pass@2 at $0.62/task (+9.75 pp over pipeline, +9.81 pp on unbiased pass@1)
  • ~75% of unique orchestrator solves utilize adaptive re-exploration
  • Removing definer think tool: −5.75 pp pass@2
  • Removing train-feedback refinement: −0.75 pp pass@2
  • Cross-model (Qwen3-235B, 99-task subset): same four-architecture lift trajectory confirmed

All results are on the ARC-AGI-1 public 400-task evaluation set; no official semi-private leaderboard score is reported.

Limitations

  • Results are on the public 400-task evaluation set only; semi-private holdout generalization is unknown and public-set scores may overestimate it.
  • No direct matched-baseline comparison with external systems (external numbers cited only as contextual reference points).
  • Cost accounting uses AtlasCloud FP8 list rates; absolute costs are deployment-specific.
  • The orchestrator's branch decision (refine / spawn / commit) is fully model-mediated with no learned branch policy or explicit confidence estimator — reliability of the branching decision is not characterized.
  • Cross-model validation is limited to a 99-task subset and one additional model.
  • Results are specific to ARC-AGI-1; generalization to ARC-AGI-2 or other abstract reasoning benchmarks is not demonstrated.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct study of harness architecture as a first-class research object: it isolates three structural levers (within-call deliberation via think tool, across-call decomposition via Explorer-Definer stages, adaptive re-exploration via mid-loop spawning) and measures each lever's independent contribution by controlled ablation on a fixed model. The diagnostic-then-test methodology — using pass@\(k\) analysis to derive a falsifiable architectural prediction and then validating it with the orchestrator — is a reusable harness-design discipline applicable beyond ARC. The generation-bound finding has direct implications for meta-harness design: it shows that upstream diversity of candidate generation is the binding constraint, not downstream selection fidelity, which should inform how budget is allocated across exploration and synthesis stages in general agentic pipelines.