Skip to content

ATHENA: Agentic Team for Hierarchical Evolutionary Numerical Algorithms

🕒 Published (v1): 2025-12-03 06:05 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ATHENA is a multi-agent agentic framework that manages the full end-to-end computational research lifecycle for Scientific Computing (SciC) and Scientific Machine Learning (SciML). Its core is the HENA loop, which formalizes the iterative research process as a Contextual Bandit problem with a knowledge-driven, no-regret policy. The system achieves validation errors on the order of \(10^{-14}\) on canonical benchmarks, outperforming direct prompting of frontier models.

Problem

Frontier foundation models cannot reliably serve as autonomous scientific researchers due to three compounding failure modes: (1) the "Lost in the Middle" phenomenon when context windows grow large, (2) autoregressive "conceptual drift" causing error accumulation over long-horizon reasoning, and (3) a lack of perceptual feedback — models cannot inspect their own outputs (loss curves, solution fields) to verify physical consistency. Prior agentic SciML frameworks either stop at ideation without implementation, optimize hyperparameters without structural diagnosis, or rely on brute-force evolutionary search that is computationally intractable when a single experiment takes hours.

Method

ATHENA operationalizes a Contextual Bandit formulation of the research lifecycle. At each step \(n\), a policy \(\pi\) selects a structural action \(A_n\) from a combinatorial action space \(\mathcal{A} = \mathcal{A}_{\text{rep}} \times \mathcal{A}_{\text{constraint}} \times \mathcal{A}_{\text{opt}}\), which is translated into executable code state \(S_n\) by an implementation operator \(I\), executed to produce multi-modal observations \(O_n\), and scored by a composite reward:

\[R_n = R_{\text{integrity}} + R_{\text{accuracy}} + R_{\text{details}} + R_{\text{optimality}}\]

The policy targets a submartingale on expected reward: \(\mathbb{E}[R_{n+1} \mid \text{History}_n] \geq R_n\).

The key mechanism preventing an intractable search is Conceptual Scaffolding — five expert-distilled blueprints (Universal Approximation, Physics-Informed ML, Operator Learning, Optimization, Numerical Methods) encoded in system prompts and base code templates. These prune the action space to mathematically valid candidates without prescribing rigid solutions.

The HENA loop has four agent groups: (A) Conceptualization (Coordinator + Gatekeeper for triage); (B) Policy (Strategy Team + Advisor Team in a proposer-critic architecture that also performs visual inspection of outputs); (C) Implementation (Code Retrieval via Agentic RAG, Implementation Team using cell-by-cell refactoring to contain conceptual drift, Inspector Agent for plan-faithfulness verification, Debugging Team); (D) Execution (runs \(S_n\), returns multi-modal \(O_n\)). Successful experiment codes are stored to grow the framework's knowledge base over time.

Key Contributions

  • Formal grounding of the agentic research loop as a Contextual Bandit with a provably submartingale-inducing policy, introducing mathematical structure absent from prior multi-agent SciML frameworks.
  • Conceptual Scaffolding mechanism that constrains the combinatorial action space using compact expert blueprints, avoiding the "Lost in the Middle" failure while preserving generative flexibility.
  • Cell-by-cell refactoring protocol in the Implementation Team that mitigates autoregressive error accumulation (conceptual drift) for codebases scaling to thousands of lines.
  • Inspector Agent that enforces \(S_n \perp S_{n-1} \mid A_n\) (Markov property) by auditing that implementations are faithful to the structural plan before execution.
  • Agentic RAG Generation for autonomously synthesizing missing code modules compatible with retrieved templates.
  • Transparent human-in-the-loop interface exposing \((A_n, S_n, O_n)\) at each iteration, enabling expert intervention that improved results by an order of magnitude in "Valley of Death" stability scenarios.
  • Demonstration of super-human performance: autonomous discovery of exact analytical solutions via Method of Characteristics and validation errors \(\sim 10^{-14}\) on canonical benchmarks where direct prompting of GPT-5.1, Gemini 3.0, and Claude Sonnet 4.5 fails.

Results

  • Validation errors of order \(10^{-14}\) on canonical SciC/SciML benchmarks.
  • On the 2D Inviscid Burgers' benchmark: GPT-5.1 and Claude Sonnet 4.5 produce "arrested motion" (incorrect Fourier Spectral methods); Gemini 3.0 produces instabilities at shock interfaces; ATHENA autonomously switches to the Method of Characteristics and recovers the exact solution.
  • Human-in-the-loop collaborative intervention improved results by an order of magnitude when the system reached a stability "Valley of Death."
  • Outperforms prior agentic SciML frameworks that explore only 15–20 total solutions via constrained evolutionary search.

Limitations

  • Scaffold blueprints are manually curated starting points and do not cover the entirety of any field; they require expert authorship and may not transfer to domains outside SciC/SciML.
  • Performance on problems requiring hours-long experiment runs is still constrained by wall-clock time, even though the approach is more sample-efficient than brute-force search.
  • The no-regret guarantee is theoretical; empirical convergence depends on the quality of the reward function and the blueprints, not a proven bound.
  • The paper truncates before showing full quantitative results tables and ablations (text provided is truncated), limiting visibility into breadth of benchmarks.
  • Currently dependent on commercial frontier APIs (GPT-5.1, Gemini 3.0, Claude 4.5, Grok 4.1); reproducibility costs may be high.

Relevance to Harnesses / Meta-Harnesses

ATHENA is a direct example of a scientific meta-harness: it orchestrates a heterogeneous team of specialized agents under a formal mathematical policy (Contextual Bandit + submartingale objective) to drive an iterative trial-execute-evaluate loop — exactly the pattern that harness/meta-harness research formalizes. The Conceptual Scaffolding mechanism is particularly relevant as a design pattern for constraining harness search spaces without hard-coding solutions, addressing the exploration-exploitation tradeoff at the harness level. The Inspector Agent enforcing plan-faithfulness and the cell-by-cell refactoring protocol represent concrete harness-layer quality controls that prevent lower-level agent outputs from drifting from high-level intent, a recurring challenge in multi-agent harness design.