Skip to content

ASI-Evolve: AI Accelerates AI

🕒 Published (v1): 2026-03-31 12:04 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ASI-Evolve is a closed-loop agentic meta-harness that automates AI research itself via a learn–design–experiment–analyze cycle. It augments evolutionary search with a cognition base (domain priors from literature injected per round) and a dedicated Analyzer that distills multi-dimensional experimental outputs into structured, reusable insights persisted to a database. It is the first unified framework to demonstrate AI-driven discovery across all three foundational AI development components: neural architectures, pretraining data pipelines, and RL training algorithms.

Problem

Existing agentic systems either tackle well-scoped tasks with direct scalar feedback (MLE-bench, AIDE, AlphaEvolve) or narrow single-pillar domains. None address the regime where all three task-complexity dimensions are high simultaneously: Execution Cost (\(C_\text{exec}\), hours of GPU training per trial, large-codebase modification), Search Space Complexity (\(S_\text{space}\), open-ended with no predefined boundaries), and Feedback Complexity (\(D_\text{feedback}\), multi-benchmark outputs, loss dynamics, efficiency traces requiring synthesis rather than scalar readout). The paper formalizes this as \(L_\text{task} = \langle C_\text{exec}, S_\text{space}, D_\text{feedback} \rangle\) and shows no prior framework occupies the high regime across all three dimensions.

Method

ASI-Evolve maintains two stores and four modules operating in each round \(t\):

  • Stores: (1) a Database \(D\) of past nodes, each recording motivation, generated program, structured results, analysis report, and metadata; (2) a Cognition Base \(C\) of domain-literature entries indexed by embeddings.
  • Round logic: Sample context nodes \(S_t \sim \text{Sample}(D)\), retrieve cognition items \(R_t = \text{Retrieve}(C; S_t)\) via semantic embedding search over \(S_t\)'s analyses/motivations, then generate a new candidate program \(p_t \sim P(p \mid S_t, R_t)\).

Four modules: 1. Researcher — LLM that produces a complete program plus natural-language motivation; supports full-code generation or diff-based incremental editing for large codebases. 2. Engineer — executes the experiment end-to-end via a user-specified evaluation script; applies wall-clock limits and lightweight quick tests for early rejection of flawed candidates; optionally invokes an LLM-as-Judge for aspects not captured by rule-based metrics. 3. Analyzer — receives the full program plus raw experimental output (logs, benchmark breakdowns, efficiency traces) and distills it into a compact, decision-oriented report that is written back to \(D\) for future retrieval; keeps context manageable without losing analytical depth. 4. Cognition — embedding-indexed store of ~150 human-prior entries (heuristics, known pitfalls, design principles from literature); injected at the start of each round to steer away from known failure modes.

Sampling policies (unified interface behind the database): UCB1, greedy, random, and MAP-Elites island algorithm — ablations show sampling policy dominates long-term trajectory quality.

Key Contributions

  • First unified demonstration of AI-driven discovery across data, architectures, and learning algorithms as a single framework.
  • \(L_\text{task} = \langle C_\text{exec}, S_\text{space}, D_\text{feedback} \rangle\) taxonomy for characterizing scientific automation difficulty.
  • Cognition base with per-round semantic retrieval to bootstrap from human priors and accelerate cold-start climb.
  • Structured Analyzer module that converts verbose multi-dimensional experimental outputs into reusable, database-persisted insights.
  • Multi-stage evaluation pipeline for architecture search: quick small-model exploration (~20M params, 2000 steps) → verification (~340M) → large-scale validation (~1.3B, 100B tokens, 16 benchmarks including 6 held-out OOD sets).
  • Evidence of cross-domain transfer (drug-target interaction, mathematics).

Results

  • Architecture (1,773 rounds, 1,350 candidates): 105 architectures surpass DeltaNet; best model achieves Dev. Avg 57.28% vs. DeltaNet 55.76% (+0.97 pts), nearly 3× the gain of the human-designed SOTA (Mamba2 +0.34 pts); generalization avg 45.40% vs. DeltaNet 44.74%.
  • Data curation: evolved pipeline improves average benchmark score by +3.96 pts; MMLU gains >18 pts over original data.
  • RL algorithm design: discovered algorithms outperform GRPO by +12.5 pts on AMC32, +11.67 pts on AIME24, +5.04 pts on OlympiadBench.
  • Circle packing (comparative benchmark): reaches SOTA-level in 17 rounds, substantially faster than OpenEvolve and GEPA.
  • Drug-target interaction (cold-start): evolved architecture achieves +6.94 AUROC improvement.

Limitations

  • Each architecture evaluation costs hours of GPU training; the framework is computationally expensive and inaccessible without large GPU clusters.
  • Large-scale validation capped at ~1.3B parameters trained on 100B tokens; scaling behavior beyond this is untested.
  • Cognition base must be seeded manually with curated literature entries per task domain; construction cost not quantified.
  • Sampling strategy requires per-task tuning (ablations show strong sensitivity); no automatic policy selection.
  • Transfer to non-AI domains described as "initial evidence" from a single biomedical experiment; generality is unconfirmed.
  • Paper text is truncated — full results for data curation and RL algorithm scenarios are not provided in the supplied excerpt.

Relevance to Harnesses / Meta-Harnesses

ASI-Evolve is structurally a meta-harness: it wraps heterogeneous sub-agents (Researcher, Engineer, Analyzer, Cognition retriever) in a deterministic orchestration loop with shared persistent state — a canonical meta-harness pattern. The Analyzer-to-database write-back, where one agent's output becomes the structured context for all future agents, directly instantiates the "insight accumulation" pattern that distinguishes meta-harnesses from flat multi-agent pipelines. The sampling abstraction (UCB1/greedy/random/MAP-Elites behind a unified interface) is essentially a harness-level scheduler, making policy choice a first-class configuration concern rather than a hard-coded loop. For researchers building harnesses, ASI-Evolve's cognition base — a retrieved, human-seeded prior injected at each orchestration step — offers a concrete design pattern for grounding expensive agentic loops in accumulated external knowledge rather than requiring cold-start exploration.