MLZero: A Multi-Agent System for End-to-end Machine Learning Automation¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MLZero is a hierarchical multi-agent system that automates end-to-end machine learning on raw multimodal data using a cognitive perception module, dual memory (semantic + episodic), and iterative code generation—all with zero human intervention. It achieves a 92% success rate on a new 25-task Multimodal AutoML Agent Benchmark, a +263.6% improvement over competitors, and outperforms all entries on MLE-Bench Lite with six gold medals.

Problem¶

Existing AutoML systems require manual preprocessing or rigid input formats and fail on raw, heterogeneous multimodal data. LLM-based ML agents suffer from (1) lack of automated data perception—forcing hard-coded preprocessing logic—and (2) hallucinated or outdated API usage when relying solely on parametric knowledge without efficient memory mechanisms.

Method¶

MLZero comprises nine specialized LLM agents organized into four modules:

Perception: A file-grouping agent analyzes raw data structure; a file-perception agent interprets file contents; a task-perception agent extracts objectives and constraints; an ML-library-selection agent maps task context to an appropriate library (e.g., AutoGluon). This produces a perceptual context P and selected library M.
Semantic Memory: Offline, a summarization agent builds queryable indices and a condensation agent distills library documentation into streamlined guidance. At each iteration, a retrieval agent fetches the top-5 condensed tutorial documents relevant to the current error context, supplying targeted external knowledge G_t to the coder.
Episodic Memory: An error-analyzer agent maintains chronological execution history. At each step it distills logs, prior code, and retrieved knowledge into a compact error summary plus suggested fix R_t, preventing token explosion from raw log accumulation.
Iterative Coding: A coder agent generates executable Python code conditioned on P, R_t, and G_t; an executer agent runs the code, captures logs, and decides whether to finalize or trigger the next debugging iteration.

The default backbone is Claude 3.7 Sonnet; the system also runs with LLaMA 3.1 8B.

Key Contributions¶

End-to-end multimodal ML automation from raw data to predictions with zero human intervention, coordinated across nine specialized agents.
Dual memory architecture: semantic memory (condensed + summary-indexed library knowledge with RAG) + episodic memory (error summarization with suggested fixes) that jointly reduce hallucination and token consumption.
New Multimodal AutoML Agent Benchmark: 25 challenging tasks spanning tabular, text, image, document, multilingual, multitable, multilabel, zero-shot, and segmentation scenarios on completely unprocessed data.
Demonstrated robustness with a compact 8B LLM (45.3% success) still outperforming all full-size baselines.
Stress-test evidence that the perception module correctly identifies task type even under deliberately misleading dataset descriptions or user instructions.

Results¶

MLE-Bench Lite (21 Kaggle tasks): MLZero success rate 86% vs. AIDE 81%, OpenHands 71%, MLAB 62%; average rank 1.43 vs. AIDE 2.36, OpenHands 2.93, MLAB 3.29; 6 gold + 2 silver medals (most among all agents).
Multimodal AutoML Agent Benchmark (25 tasks):
MLZero (default): success rate 92.0%, avg rank 2.42
AIDE (default): 25.3% success, avg rank 6.16
Codex CLI (default): 45.3% success, avg rank 5.76; with reasoning LLM (+rea): 69.3% success, avg rank 2.83
DS-Agent: 14.7% success
AutoKaggle: 13.3% success
Token/cost efficiency: MLZero uses 63k tokens at $0.19/dataset vs. AIDE's 273k tokens at $0.82—4.3× fewer tokens, 23% of the cost.
Ablation – semantic memory: removing external knowledge drops to 69.3% success; retrieving 5 documents is optimal; condensation reduces tokens without performance loss.
Ablation – episodic memory: removing error summaries + fixes drops to 83% success; removing only fixes drops to 75% (worst) due to increased confused iterations.
Error analysis: MLZero overall failure rate 6.5% vs. Codex CLI+rea 26.9%, AIDE 47.8%, DS-Agent 76.0%; MLZero eliminates perception, postprocessing, and API-hallucination error categories entirely.
8B LLM: 45.3% success, avg rank 5.14—still outperforms all baselines at default settings.

Limitations¶

Benchmark coverage is limited by the set of integrated ML libraries; tasks requiring diffusion models or specialized audio models fail because those libraries are not registered.
3-hour per-task wall-clock limit constrains solution quality; the 24-hour extended run approaches but does not fully match expert human performance.
The 8B configuration degrades significantly in success rate (45.3%), indicating sensitivity to backbone capability for complex tasks.
MLE-Bench Lite uses preprocessed data (not fully raw), which may overestimate agent capabilities in real-world scenarios.
Environmental cost of running large LLMs for each task is non-trivial; the paper acknowledges this but does not quantify emissions.
Stress tests cover only a single tabular regression task (Abalone); resilience on harder multimodal tasks under noisy inputs is not evaluated.

Relevance to Agentic AI / LLM Agents¶

MLZero is a concrete instantiation of the hierarchical multi-agent paradigm applied to a real-world automation domain, demonstrating how specialized agents with distinct roles (perception, retrieval, error analysis, coding, execution) outperform monolithic LLM pipelines on complex long-horizon tasks. The dual-memory design—semantic for domain knowledge retrieval and episodic for error-state tracking—directly addresses core failure modes of LLM agents (hallucination and context bloat), offering a reusable architectural pattern. The new Multimodal AutoML Agent Benchmark provides a rigorous end-to-end evaluation setting that sidesteps the preprocessed-data shortcoming of MLE-Bench, advancing how the community measures true agent autonomy. The finding that an 8B agent system outperforms full-scale single-agent baselines reinforces that agent architecture and memory design matter as much as raw model scale.