Skip to content

ARCANA: A Reflective Multi-Agent Program Synthesis Framework for ARC-AGI-2 Reasoning

🕒 Published (v1): 2026-07-10 03:03 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent synthesis framework with iterative perception-hypothesis-execution loop; ARC-AGI-2 scale

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ARCANA is a four-agent collaborative framework that casts each ARC-AGI-2 task as a multi-turn sequential decision episode coordinated by a learned Meta-Controller and a shared differentiable blackboard. A perception agent builds object-centric scene graphs, a CVAE-based agent proposes diverse DSL programs, a symbolic executor verifies them, and a reflective agent synthesizes failure-driven feedback to steer the next hypothesis generation turn. Under ARC Prize 2026 hardware constraints (4×L4, 12 h, offline), ARCANA achieves 32.5% accuracy on the semi-private evaluation set, +6.5 pp over the prior strongest baseline.

Problem

ARC-AGI-2 demands inferring compact grid-transformation rules from only a few demonstrations while handling variable grid sizes, object interactions, and a high-ambiguity latent rule space. Chain-of-thought prompting exposes reasoning steps but does not guarantee executable consistency; iterative self-feedback is too weak when the hypothesis space is symbolic, spatial, and constrained to exact pixel-level outputs. Existing neural transductive and standalone program-synthesis methods lack an adaptive mechanism to diagnose failure patterns and redirect search within a tight compute budget.

Method

ARCANA decomposes each task into a multi-turn episode \(E = (s_0, a_1, s_1, \ldots, s_T)\) across four agents:

Perceptual Grounding Agent (PGA): A 6-layer Transformer with factored 2D-RoPE encodes each grid cell; \(K=16\) differentiable Slot Attention slots discover objects. Pairwise delta representations across demonstration pairs are aggregated by a 2-layer cross-demonstration Transformer into a task embedding \(R_\text{task} \in \mathbb{R}^{d_r}\).

Hypothesis Generation Agent (HGA): A CVAE with conditional prior \(p_\theta(z_\pi | R_\text{task}, F_\text{refine}) = \mathcal{N}(\mu_\theta(c), \text{diag}(\sigma^2_\theta(c)))\) samples latent program codes; a 4-layer causal Transformer decoder auto-regressively generates programs over a 47-primitive DSL. At inference, \(M\) codes are drawn and a \(k\)-DPP selects a maximally diverse subset.

Symbolic Execution Agent (SEA): Executes candidate programs on demonstrations, computing a hierarchical score \(V(\pi^{(m)}, D_\tau)\) combining exact match, cell accuracy, and SSIM (weights \(\gamma_1=0.3\), \(\gamma_2=0.1\)). A BiGRU summarizes per-step traces into \(e_\text{trace}^{(m)}\) written to the blackboard.

Reflective Refinement Agent (RRA): Counterfactual credit assignment blames step \(t\) via \(\rho_t^{(m)} = V(\pi^{(m)}, D_\tau) - V(\pi^{(m)}_{\setminus t}, D_\tau)\); a residual MLP fuses error, blame-weighted primitives, and trace into per-candidate feedback, aggregated via attention into \(F_\text{refine}\), which conditions the HGA prior away from failed program regions.

Meta-Controller: A GRU tracks state over the blackboard contents and best-so-far verification score. A multi-label Bernoulli policy fires agents per turn; per-turn compute \(b_{t+1} = C_{\text{rem},t}\cdot\sigma(w_b^\top s_t + b_b)\) enforces a \(\$0.42\)/task ceiling. Termination fires on full verification, improvement plateau (\(\Delta V_t < \eta_\text{stop}\)), or budget exhaustion.

Test-Time Adaptation (TTT-LoRA): Before the agentic loop, 50 AdamW steps fine-tune LoRA adapters (\(r=8\)) on the PGA attention weights using the demonstration pairs with D8 augmentation.

Training: End-to-end Reasoning Trajectory Optimization (RTO): \(\mathcal{L}_\text{total} = \lambda_1 \mathcal{L}_\text{focal} + \lambda_2 \mathcal{L}_\text{VAE} + \lambda_3 \mathcal{L}_\text{PG} + \lambda_4 \mathcal{L}_\text{contra}\) with \(\lambda_{1:4} = (1.0, 0.5, 0.1, 0.3)\); KL annealed \(0\to1\) over 10K steps; REINFORCE policy gradient rewards both final accuracy and turn efficiency.

Key Contributions

  • Four-agent architecture (PGA → HGA → SEA → RRA) with a shared differentiable blackboard enabling structured multi-turn refinement.
  • RRA with counterfactual step-level credit assignment and attention-based feedback synthesis that steers the CVAE prior at the latent level.
  • Learned Meta-Controller (GRU + multi-label Bernoulli policy) with explicit per-turn compute allocation and learnable stopping threshold.
  • CVAE + \(k\)-DPP diverse sampling for maximally non-redundant candidate program generation.
  • Reasoning Trajectory Optimization (RTO): a four-term training objective jointly optimizing answer correctness, program likelihood, policy efficiency, and contrastive refinement signal.
  • TTT-LoRA: lightweight 50-step test-time adaptation of the PGA using only demonstration pairs.

Results

All constrained methods: 4×NVIDIA L4 GPUs, 12 h, offline. Semi-private evaluation set: 120 tasks.

  • ARCANA: 32.5% accuracy, 67.4% CellAcc, CNE 7.90, $0.16/task — best among constrained open-source methods.
  • Full ablated configuration: 35.8% accuracy, 69.2% CellAcc.
  • Strongest prior baselines: EPS-Grok 26.0%, NVARC 24.0%, SOAR 18.5%.
  • Human baseline: 75.0% accuracy, 94.1% CellAcc.
  • Pass@k (ARCANA): @2 = 35.8%, @8 = 45.0%, @32 = 53.3%, @128 = 59.2% — steeper curve than SOAR/NVARC, confirming better candidate diversity.
  • Ablations (accuracy drop vs. full 35.8%):
  • −RRA: 25.0% (−10.8 pp) — largest single-component impact.
  • −TTT-LoRA: 28.3% (−7.5 pp).
  • −DPP diverse sampling: 31.7% (−4.1 pp).
  • −Meta-Controller (fixed schedule): 33.0% (−2.8 pp).
  • −CVAE latent space: 33.4% (−2.4 pp).
  • −Cross-Demo Transformer: 34.2% (−1.6 pp).
  • Average 3.7 refinement turns; \(V_\text{best}\) rises monotonically: 0.42 → 0.58 → 0.67 → 0.71.
  • Largest gains over EPS-Grok: multi-rule composition (+8.8%) and multi-step sequential tasks (+8.9%).

Limitations

  • Substantial gap to human performance persists (~42.5 pp in accuracy, ~24.9 pp in CellAcc).
  • Symbolic interpretation tasks remain hardest across all methods, suggesting the DSL and search strategy are insufficient for that sub-category.
  • Only 48M parameters; performance under larger backbones not evaluated.
  • Full-configuration cost per task not reported; only ablation/comparison costs given for constrained setting.
  • Evaluation restricted to ARC Prize 2026 hardware protocol — generalizability to other compute regimes unknown.
  • Slot Attention with \(K=16\) fixed slots may fail on tasks with more than 16 objects.

Relevance to Harnesses / Meta-Harnesses

ARCANA's Meta-Controller is a canonical learned harness: a GRU-based orchestrator that decides which sub-agents to activate each turn, allocates compute per turn against a global budget ceiling, and fires a learnable stopping criterion — all central harness design questions. The shared differentiable blackboard acts as the harness's communication substrate, with agents reading and writing typed key-value entries via scaled-dot-product attention, making it a differentiable analogue of the structured context-passing seen in handcrafted harnesses. The RTO training objective directly optimizes the harness-level behavior (efficiency of the multi-turn episode, not just final-answer correctness), providing a principled method for training harness policies end-to-end rather than engineering them manually. For researchers building or evaluating meta-harnesses, ARCANA demonstrates that budget-aware, failure-conditioned agentic loops with learned scheduling outperform fixed-pipeline alternatives by a substantial margin, and that reflective feedback routed back to the proposal stage (−10.8 pp on removal) is the highest-leverage single harness feature.