Cost-Effective Agent Harnesses for Abstract Reasoning and Generalization on ARC-AGI-1¶

🕒 Published (v1): 2026-07-07 19:49 UTC · Source: Arxiv · link

Why this paper was selected

Directly about cost-effective harnesses; competitive on ARC-AGI-1 without frontier-scale compute

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper studies a third regime for ARC-AGI-1 beyond heavy test-time compute and benchmark-specific training: agentic harness engineering on an open-weight model (DeepSeek V3.2, non-thinking mode) under a strict sub-$1 budget. Two novel architectures — an Explorer-Definer Pipeline and a Reflective Orchestrator — lift a 15.50% one-shot baseline by ~52 percentage points to 67.25% pass@2 at $0.62/task, with no ARC-specific fine-tuning.

Problem¶

Strong ARC-AGI-1 performance has required either frontier-model evolutionary search ($8–$400/task) or benchmark-specific fine-tuning. The region below $1/task with a general-purpose open-weight model and no ARC-specific training is largely unoccupied above ~30% accuracy. It is unclear how much of the gap between cheap one-shot LLM calls (~16%) and competitive systems (≥50%) is attributable purely to architectural design choices versus model capability or training distribution.

Method¶

All architectures run DeepSeek V3.2 in non-thinking mode on the ARC-AGI-1 public 400-task evaluation set. Four conditions of increasing structural sophistication are compared:

One-shot baseline: single LLM call, structured JSON output only.
CoT baseline: adds a reasoning field before output in the JSON schema to force externalized deliberation.
Explorer-Definer Pipeline: two sequential phases. Phase 1 runs $N$ parallel PatternExplorer agents, each equipped with a think(thought) scratchpad tool and a note_pattern(pattern) logging tool; they independently analyze training pairs and emit compressed natural-language pattern descriptions. Phase 2 runs a TransformationDefiner agent that reads all explorer findings and synthesizes a Python transformation function; $M$ candidates are generated via best-of-sampling, verified by exact execution on training pairs, and deduplicated by test-prediction. Selection uses training-pair accuracy.
Reflective Orchestrator: augments the definer with a mid-loop tool to spawn $K=2$ fresh focused explorers conditioned on a structured failure summary when the current hypothesis fails particularly badly on training pairs. The orchestrator controls its own loop-exit criteria.

The canonical pipeline configuration is $N=5$, $t=0.5$ (explorer temperature), $M=5$ (definer candidates). Costs are computed from raw token counts at $\$0.30$/M prompt and $\$0.38$/M completion. A Pareto sweep over $N \in \{1,2,5\} \times t \in \{0.0, 0.5, 1.0\} \times M \in \{1,...,5\}$ characterizes the cost–accuracy surface. The unbiased pass@k estimator

\[\widehat{\text{pass@k}} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\]

is used to diagnose whether the pipeline is generation-bound or selection-bound.

Key Contributions¶

Explorer-Definer Pipeline: 57.50% pass@2 at $0.25/task (+42 pp over one-shot, +27.5 pp over CoT).
Reflective Orchestrator: 67.25% pass@2 at $0.62/task (+9.75 pp over pipeline; 95% CI [+6.50, +13.25]).
Generation-bound diagnostic: unbiased pass@k analysis shows training-pair selection captures ~95% of the candidate ceiling, establishing the bottleneck is generation diversity, not selection quality.
Orchestrator as architectural test: unbiased pass@1 lift (+9.81 pp) matches naive pass@2 lift (+9.75 pp), confirming the gain is generation-side rather than selection-mediated.
Ablations: removal of the definer's think scratchpad tool costs −5.75 pp pass@2; removal of train-feedback refinement costs only −0.75 pp.
Cross-model validation: the same four-architecture lift trajectory holds on Qwen3-235B over a 99-task subset, suggesting the harness rather than the specific model drives the gains.
Methodological discipline: token-normalized cost reporting, explicit pass@k definitions, bootstrap CIs, and paired-bootstrap deltas throughout.

Results¶

One-shot baseline: 15.50% pass@2
CoT baseline: 30.00% pass@2
Explorer-Definer Pipeline ($N=5, t=0.5, M=5$): 57.50% pass@2 at $0.25/task
Reflective Orchestrator ($N=5, t=0.5, M=5$): 67.25% pass@2 at $0.62/task (+9.75 pp over pipeline, 95% CI [+6.50, +13.25])
Unbiased pass@1 lift from pipeline → orchestrator: +9.81 pp (confirming generation-side gain)
Selection via training-pair accuracy captures ~95% of the candidate ceiling (pipeline is generation-bound)
~75% of orchestrator unique solves utilize adaptive re-exploration
Pareto sweep: explorer count $N$ is the dominant cost–accuracy lever; best-of-$M$ sampling saturates; intermediate temperature ($t=0.5$) often dominates
Disabling definer think tool: −5.75 pp pass@2; disabling train-feedback refinement: −0.75 pp
Reference context: Berman 2025 reaches 79.6% at $8.42/task (frontier evolutionary search); Pang reaches 77.1% at $2.56/task

Limitations¶

Evaluated only on the ARC-AGI-1 public 400-task evaluation set; no official semi-private leaderboard score, so results may overestimate semi-private generalization.
No direct matched baselines against external systems (reported only as contextual reference points).
Cross-model validation on Qwen3-235B covers only a 99-task subset, limiting statistical power.
The orchestrator's branch decision (refine, spawn, or commit) is model-mediated with no explicit learned branch policy or confidence estimator, which may be suboptimal.
Cost figures are tied to AtlasCloud FP8 list rates; self-hosting or batch discounts could change absolute but not relative ordering.
Performance ceiling relative to frontier-model regimes (79.6%) remains substantial; the approach has not been tested on ARC-AGI-2.

Relevance to Agentic AI / LLM Agents¶

This paper directly advances the design science of LLM agent harnesses, demonstrating that decomposing a complex reasoning task into explicit exploration and synthesis stages — with adaptive mid-loop re-exploration — yields dramatic, cost-efficient performance gains without any model specialization. The generation-bound vs. selection-bound diagnostic is a general methodology for identifying architectural bottlenecks in multi-sample agent pipelines, applicable beyond ARC. The Reflective Orchestrator's pattern — spawning upstream agents conditioned on structured failure signals when downstream synthesis stalls — is a concrete, reusable template for self-correcting agentic loops. The work also contributes rigorous cost-accounting and pass@k methodology that the broader agent-harness literature often lacks.