Skip to content

MLZero: A Multi-Agent System for End-to-end Machine Learning Automation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MLZero is a nine-agent LLM-driven system that automates end-to-end multimodal ML pipelines—from raw unprocessed data to prediction outputs—with zero human intervention. It combines a cognitive perception module with dual semantic/episodic memory to guide iterative code generation and debugging. On the Multimodal AutoML Agent Benchmark (25 tasks), it achieves a 0.92 success rate, outperforming baselines by +263.6%.

Problem

Existing AutoML systems require hard-coded preprocessing logic, rigid input formats, or expert-defined workflows, and fail to handle truly raw multimodal data end-to-end. LLM-based agents further suffer from hallucinated code generation, outdated API knowledge, and inefficient memory management under long, convoluted ML contexts.

Method

MLZero orchestrates nine specialized LLM agents across four modules:

  1. Perception (\(P(x, U^{opt}) = (P, M)\)): Three agents (file grouping, file perception, task perception) parse arbitrary raw data structure and semantics; an ML Library Selection agent maps task characteristics to the appropriate ML library \(M\).

  2. Semantic Memory (\(S_t(P, M, R_t) = G_t\)): Offline, a Summarization agent and Condensation agent compress library documentation into queryable indices. At each iteration \(t\), a Retrieval agent fetches condensed tutorials \(G_t\) conditioned on current error context \(R_t\).

  3. Episodic Memory (\(E_t(P, C_{t-1}, L_{t-1}, G_{t-1}, R_{t-1}) = R_t\)): An Error Analyzer agent distills execution logs, prior code artifacts, and prior error context into a compact error summary + suggested fix \(R_t\), avoiding token explosion over long histories.

  4. Iterative Coding (\(\mathcal{G}_t(P, U_t^{opt}, R_t, G_t) = (y_t, C_t, L_t)\)): A Coder agent synthesizes perceptual context, episodic error context, and retrieved semantic knowledge into executable code \(C_t\); an Executer agent interprets logs \(L_t\) and decides success or triggers iteration \(t+1\).

The system uses Claude 3.7 Sonnet by default, with a 3-hour per-dataset time budget.

Key Contributions

  • Nine-agent hierarchical system delivering zero-intervention end-to-end AutoML on raw multimodal data
  • Cognitive perception module that dynamically selects ML libraries and generates structured task context without hard-coded preprocessing
  • Dual memory architecture: semantic memory for retrieved library knowledge, episodic memory for compressed debugging history, jointly reducing hallucinations and token consumption
  • Multimodal AutoML Agent Benchmark: 25 tasks spanning tabular, text, image, document, multilingual, multitable, multilabel, and zero-shot scenarios (harder than MLE-Bench Lite's preprocessed inputs)
  • Demonstrated robustness with a compact 8B LLaMA 3.1 backbone (45.3% success vs. baselines) versus full-size systems

Results

  • Multimodal AutoML Agent Benchmark (25 tasks):
  • MLZero (default): success rate 0.92, avg. rank 2.42
  • AIDE: success rate 0.253, avg. rank 8.04
  • AutoKaggle: success rate 0.693, avg. rank 6.16
  • DS-Agent: success rate 0.147, avg. rank 8.26
  • Codex CLI: success rate 0.693, avg. rank 6.02
  • MLZero vs. best baseline: +263.6% success rate
  • MLE-Bench Lite (21 Kaggle tasks):
  • MLZero: success rate 86%, avg. rank 1.43, 6 gold + 2 silver medals
  • AIDE: 81% success, rank 2.36
  • OpenHands (CodeActAgent): 71% success, rank 2.93
  • MLAB (ResearchAgent): 62% success, rank 3.29
  • 8B ablation: 45.3% success rate, rank 5.14—outperforms DS-Agent (14.7%) and AIDE (25.3%)
  • 24hr extended run: approaches human-reported performance levels on benchmark tasks

Limitations

  • Three-hour per-dataset time budget may be insufficient for some complex tasks; performance improves substantially in the 24-hour extended setting
  • ML library coverage constrains solvable tasks (e.g., diffusion models, audio-specific libraries not yet registered)
  • 8B backbone degrades substantially (45.3% vs. 92.0%), indicating strong backbone LLM dependency
  • MLE-Bench Lite uses preprocessed data, so results there may overestimate real-world difficulty relative to the multimodal benchmark
  • No evaluation of iterative human-in-the-loop configurations (per-iteration user input disabled in all experiments)
  • Failure case analysis is qualitative; systematic categorization of error types is left to ablations

Relevance to Harnesses / Meta-Harnesses

MLZero is a direct exemplar of a production-grade multi-agent meta-harness: it meta-orchestrates nine specialized sub-agents across perception, memory, coding, and execution phases, with control flow determined by runtime feedback rather than static pipelines. The dual-memory architecture (semantic + episodic) is a harness-level design pattern for managing context across iterative agent loops—precisely the problem harness builders face when chaining LLM calls over long, stateful workflows. The offline knowledge condensation pipeline (Summarization + Condensation agents) mirrors how meta-harnesses pre-process and cache domain knowledge for downstream agents, reducing redundant retrieval. The iterative coding loop with error-context injection and execution-feedback routing is a clean reference implementation of the harness "retry-with-context" pattern.