Skip to content

TusoAI: Agentic Optimization for Scientific Methods

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Agentic system automates iterative scientific method development and testing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TusoAI is an agentic AI system that autonomously develops and optimizes computational methods for scientific tasks by combining a domain knowledge tree with hierarchical Bayesian planning and iterative diagnostic feedback. It takes a task description and evaluation function as input, runs for an 8-hour budget, and outputs an optimized Python implementation. Across 11 benchmarks (6 single-cell, 5 scientific deep learning), TusoAI achieves average ranks of 1.2 and 2.8 respectively, consistently outperforming expert methods and MLE agents.

Problem

Scientific computational method development is bottlenecked by the need to iteratively review literature, test domain-specific modeling assumptions, and implement efficient code—a cycle that takes expert teams years. Existing LLM-based scientific agents (Biomni, ChemCrow) execute pipelines of existing tools rather than inventing new algorithms. General MLE agents (AIDE, DS-Agent, MLE-STAR) optimize over predefined ML search spaces and cannot handle the unstructured, continually evolving domain knowledge and custom evaluation metrics inherent in scientific research.

Method

TusoAI takes task description \(T\), dataset \(D\), evaluator \(h(\cdot): S_\text{full} \to \mathbb{R}\), and optional warm-start solution \(s_\text{init}\), and outputs \(s^* = \arg\max_{s \in S_\text{full}} h(s)\) within time budget \(T_\text{budget}\) (default 8 hours).

Step 1 — Knowledge gathering: Retrieves up to 10 papers from Semantic Scholar (ranked by citation count). An agent \(A_\text{paper}\) creates a 15-point technical summary per paper, iteratively refined using each paragraph of the Methods section (capped at 1,200 words/paper).

Step 2 — Knowledge tree construction: A draft-then-refine strategy builds a two-level tree: (1) optimization categories \(C\) (e.g., "regularization", "single-cell noise modeling") each with probability \(\pi_c\)—higher weights assigned to earlier-pipeline categories—and (2) per-category instruction lists \(I_c\) (10 base + 10 per paper summary). A predefined diagnostic category \(I_\text{diag}\) guides data/model logging.

Step 3 — Iterative optimization: Initializes 5 candidate solutions; each subsequent iteration selects diverse top performers via code-text cosine similarity clustering (shortest solution within 0.1% of cluster best). Per top solution \(s\): - With probability \(\alpha=0.8\): instruction-based optimization — samples \(c \sim \text{Cat}(\{\pi_c\})\), draws 3 candidate instructions, selects most promising, generates \(s'\). If \(h(s') > h(s)\): \(\pi_c \leftarrow 1.1\pi_c\) (Bayesian update); feedback agent appends \(A_\text{feedback}(s, s') \to F_c\). - With probability 0.2: diagnostic-based optimization — runs \(s\) to collect logs, then generates \(s'\) guided by \(I_\text{diag}\) and \(F_\text{diag}\).

Each implementation attempt is time-capped (10 min for init, 10 min for iteration) with limited bug-fix retries (4 and 2 respectively), enforcing scalability.

Key Contributions

  • TusoAI agent architecture specifically designed for scientific method development (not execution of existing tools)
  • Knowledge tree: two-level structure (categories → instructions) grounded in paper summaries, enabling structured, diverse exploration
  • Hierarchical Bayesian planning: adaptive category sampling with multiplicative utility updates (\(\pi_c \leftarrow 1.1\pi_c\) on success) balancing exploitation and diversity
  • Diagnostic-based optimization: dedicated loop that instruments running solutions, collects intermediate data/model statistics, and uses them to guide refinement
  • Benchmark suite across 6 single-cell analysis tasks (OpenProblems) and 5 scientific deep learning tasks (NASBENCH-360)
  • Two genetics case studies where TusoAI improved state-of-the-art and uncovered new biology

Results

Single-cell benchmarks (Table 1, 3 seeds, vs. Expert / AIDE / Biomni / ChatGPT-Agent): - TusoAI avg rank 1.2 vs. AIDE 3.0, Expert 3.7, ChatGPT-Agent 3.5 - TusoAI avg performance 0.66 vs. Expert 0.57, AIDE 0.52, Biomni 0.49 - Highest individual scores: Denoise 0.35 (Expert 0.28), SVG 0.80 (Expert 0.66), Batch 0.83 (tied with ChatGPT-Agent)

Scientific deep learning benchmarks (Table 2, vs. NAS methods, expert models, MLE agents): - TusoAI avg rank 2.8 vs. AIDE 9.8, Expert 4.6, DARTS 4.0, Claude-4-Sonnet 4.6 - TusoAI avg performance 0.70 vs. Expert 0.69, DARTS 0.71 (which has N/A on AMBER task) - Top performance on Satellite (0.89) and DeepSEA (0.70, matching AMBER)

Diversity: TusoAI generates substantially higher code diversity than AIDE (measured by \(1 - \text{cosine similarity}\) of text embeddings) throughout the 8-hour window, linked to stronger optimization trajectories.

Efficiency: All generated methods run in \(<3\) min (single-cell) and \(<8\) min (deep learning).

Ablations: Removing any of {categories, Bayesian sampling, diagnosis, domain knowledge} each degrades performance; full system is strictly best.

Limitations

  • Fixed 8-hour wall-clock budget per task; no assessment of performance scaling with time
  • Paper retrieval capped at 10, ranked by citation count—may miss recent or domain-niche work
  • Operates on a single function within a larger codebase; cannot redesign full pipelines
  • Warm-start setting still requires a working initial implementation \(s_\text{init}\)
  • Evaluation relies on task-specific evaluators \(h(\cdot)\) provided externally; quality of evaluation function is not addressed
  • Genetics case study results are truncated in the provided text, limiting full assessment of biological novelty claims

Relevance to Agentic AI / LLM Agents

TusoAI represents a concrete instantiation of the "AI scientist" paradigm: an agent that closes the loop between literature retrieval, structured knowledge representation, hypothesis generation (as code), empirical evaluation, and feedback-driven refinement—all without human intervention. Its knowledge tree with Bayesian category sampling is a transferable architectural pattern for any agentic system that must explore a large, heterogeneous action space with varying utility. The diagnostic-based optimization loop—where an agent instruments its own outputs to guide next-step decisions—is an underexplored but practically valuable pattern for agentic code optimization beyond scientific domains. The benchmark results directly establish that domain-knowledge integration and diversity-promoting search are critical missing ingredients in general-purpose MLE agents like AIDE.