Skip to content

ARCANA: A Reflective Multi-Agent Program Synthesis Framework for ARC-AGI-2 Reasoning

🕒 Published (v1): 2026-07-10 03:03 UTC · Source: Arxiv · link

Why this paper was selected

Reflective multi-agent program synthesis on ARC-AGI-2; hard public benchmark with constrained test-time compute

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ARCANA is a four-agent collaborative framework for ARC-AGI-2 that decomposes abstract grid-transformation reasoning into perception, hypothesis generation, symbolic execution, and reflective refinement, coordinated by a learned meta-controller over a shared differentiable blackboard. With only 48M parameters under ARC Prize 2026 hardware constraints, it achieves 32.5% accuracy on the semi-private evaluation set, outperforming the next best constrained method (EPS-Grok at 26.0%) by 6.5 points.

Problem

ARC-AGI-2 requires inferring compact transformation rules from few demonstrations over variable-size grids with severe ambiguity in the rule space. Pure neural approaches lack executable consistency, chain-of-thought prompting fails to guarantee correct grid outputs, and iterative text-based self-refinement is too weak when the hypothesis space is symbolic and spatially constrained.

Method

ARCANA casts each task as a sequential decision-making episode with four specialized agents:

  1. Perceptual Grounding Agent (PGA): A 6-layer Transformer with factored 2D RoPE encodes grid cells, then differentiable Slot Attention (\(K=16\) slots, \(R=3\) iterations) discovers objects unsupervised. A cross-demonstration Transformer aggregates per-pair delta representations into a permutation-invariant task representation \(R_\text{task} \in \mathbb{R}^{d_r}\).

  2. Hypothesis Generation Agent (HGA): A CVAE with conditional prior \(p_\theta(z_\pi | R_\text{task}, F_\text{refine})\) samples diverse latent codes decoded auto-regressively into programs over a 47-primitive DSL. A \(k\)-DPP selects a maximally diverse candidate subset balancing quality and coverage.

  3. Symbolic Execution Agent (SEA): Evaluates candidates compositionally on demonstrations using a hierarchical score combining exact-match, cell accuracy, and SSIM: \(V(\pi^{(m)}, D_\tau) = \frac{1}{N}\sum_n [\mathbb{1}[\hat{G}_n^{(m)}=G_n^\text{out}] + \gamma_1 \text{CellAcc}_n + \gamma_2 \text{SSIM}]\) with \(\gamma_1{=}0.3, \gamma_2{=}0.1\). Execution traces are encoded by a BiGRU and written to the blackboard.

  4. Reflective Refinement Agent (RRA): Performs counterfactual credit assignment—blaming step \(t\) via \(\rho_t^{(m)} = V(\pi^{(m)}, D_\tau) - V(\pi^{(m)}_{\setminus t}, D_\tau)\)—and synthesizes feedback \(F_\text{refine}\) that conditions the HGA prior to steer sampling away from failed program regions.

A GRU-based Meta-Controller decides per turn which agents fire and how much compute to allocate, enforcing a \(\$0.42\)/task ceiling via a budget-aware Bernoulli activation policy. Test-time adaptation fine-tunes LoRA adapters (rank \(r=8\)) on PGA attention weights for 50 AdamW steps using D8 symmetry augmentation. Training uses a Reasoning Trajectory Optimization (RTO) objective: \(\mathcal{L}_\text{total} = \lambda_1 \mathcal{L}_\text{focal} + \lambda_2 \mathcal{L}_\text{VAE} + \lambda_3 \mathcal{L}_\text{PG} + \lambda_4 \mathcal{L}_\text{contra}\) with \(\lambda_{1:4} = (1.0, 0.5, 0.1, 0.3)\).

Key Contributions

  • Four-agent architecture with role specialization (perception → hypothesis → execution → reflection) over a shared differentiable blackboard
  • Counterfactual credit assignment in RRA that identifies harmful DSL primitives and synthesizes gradient-like feedback to reshape the HGA latent prior
  • CVAE + k-DPP hypothesis generation that jointly maximizes diversity and program quality
  • Learned Meta-Controller with budget-aware per-turn agent activation under hard compute constraints (\(\$0.42\)/task ceiling)
  • Test-time LoRA adaptation of perceptual grounding with D8 symmetry augmentation (50 steps, no full retrain)
  • RTO training objective combining focal reconstruction, CVAE ELBO, REINFORCE policy gradient, and InfoNCE contrastive loss

Results

  • 32.5% accuracy on semi-private ARC-AGI-2 evaluation set (120 tasks); 35.8% with full configuration (ablation set); human baseline 75.0%
  • CNE = 7.90 (cost-normalized efficiency), best among all constrained methods
  • +6.5% over EPS-Grok (26.0%), +8.5% over NVARC (24.0%), the next strongest constrained baselines
  • CellAcc 67.4% vs. EPS-Grok 58.2%, NVARC 56.8%
  • Pass@128 = 59.2% vs. EPS-Grok 50.0%, SOAR 46.7%
  • Ablations: removing RRA costs −10.8%; removing TTT-LoRA costs −7.5%; removing DPP costs −0.8%; removing CVAE costs −2.4%; removing cross-demo Transformer costs −1.6%
  • Largest gains over EPS-Grok on multi-rule composition (+8.8%) and multi-step sequential tasks (+8.9%); symbolic interpretation hardest for all methods
  • Average 3.7 refinement turns; \(V_\text{best}\) rises monotonically across turns (0.42 → 0.58 → 0.67 → 0.71)
  • Cost: \(\$0.16\)/task at 48M parameters

Limitations

  • Substantial gap to human performance persists (32.5% vs. 75.0%)
  • Symbolic interpretation tasks remain hardest for all evaluated methods, including ARCANA
  • Evaluation constrained to a 120-task semi-private set; generalization to the full ARC-AGI-2 distribution is not established
  • Test-time adaptation (TTT-LoRA) is task-specific and adds per-task fine-tuning overhead
  • No reported comparison against larger unconstrained models (e.g., frontier LLMs with no hardware ceiling)
  • The paper does not characterize failure modes qualitatively or analyze which task types remain unsolved

Relevance to Agentic AI / LLM Agents

ARCANA exemplifies a core design pattern in agentic AI: decomposing a hard reasoning task into specialized sub-agents with distinct roles, coordinated by a learned orchestrator that manages compute budgets dynamically. The RRA's counterfactual credit assignment and feedback synthesis—steering subsequent hypothesis generation via a differentiable signal rather than natural language—represents a concrete, executable alternative to text-based self-refinement, directly addressing a known weakness of chain-of-thought and self-refine approaches in symbolic domains. The shared differentiable blackboard and Meta-Controller architecture are directly transferable patterns for multi-agent LLM systems that must operate under latency or cost constraints. The CVAE + k-DPP diverse program proposal loop also illustrates how structured exploration can be maintained in agentic search without collapsing to a single high-probability trajectory.