MLE-Smith: Scaling MLE Tasks with Automated Multi-agent Pipeline¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Percy Liang (Stanford); scalable multi-agent pipeline for ML engineering tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MLE-Smith is a fully automated multi-agent pipeline that transforms raw datasets into competition-style machine learning engineering (MLE) tasks via a generate–verify–execute paradigm, removing the human-curation bottleneck that limits existing MLE benchmarks. It produced 606 verified tasks from 224 Kaggle datasets at ~$0.78/task, with LLM rankings on generated tasks strongly correlating with rankings on human-curated benchmarks.

Problem¶

Existing MLE agent benchmarks (MLE-Bench, DS-Bench, MLE-Dojo, MLGym) rely on static, manually curated task sets that require significant expert effort to design competitions and engineer standardized evaluation pipelines. This creates a scalability bottleneck: task diversity and volume are fundamentally capped by human throughput, limiting the data available for training and evaluating next-generation MLE agents.

Method¶

MLE-Smith pipelines three specialized agents sequentially:

Brainstormer: Given a raw dataset and tool access (file I/O, shell, code execution), it enumerates up to 3 candidate task formulations per dataset—specifying prediction targets, evaluation metrics, and data utilization rationale—without committing to a single design.
Designer: For each candidate, instantiates a fully specified MLE task package: deterministic train/test splits, input/output schemas, evaluation protocol, metric scripts (metric.py), preparation scripts (prepare.py), sample submissions, and test scripts.
Refactor: Standardizes all artifacts into a unified canonical schema (consistent function signatures, file hierarchy raw/, private/, public/, metric.py, prepare.py) enabling automated downstream validation.

Quality is enforced by a Hybrid Verification Mechanism with three layers: - Assertions (deterministic): hard checks on file existence, directory layout, script executability, and schema conformance—before and after Refactor. - Reviews (LLM-based): semantic validation for task clarity, metric appropriateness, and absence of ground-truth leakage. - Execution-based validation: a ReAct-style MLE agent with a 10-step budget runs the full pipeline inside an MLE-Dojo environment, confirming non-trivial predictive performance and metric sensitivity.

Failures at any layer are routed back to Designer or Refactor for targeted refinement.

Key Contributions¶

First end-to-end automated framework for generating competition-style MLE tasks from raw datasets without human intervention.
A three-layer hybrid verification mechanism (Assertions + Reviews + Execution) guaranteeing structural integrity, semantic soundness, and empirical solvability.
A corpus of 606 fully verified MLE tasks from 224 Kaggle datasets spanning tabular, vision, audio, time series, NLP, and video modalities, with diverse objectives (classification, regression, ranking, generation) and metrics.
Empirical demonstration that LLM rankings on MLE-Smith tasks strongly correlate with rankings on human-curated MLE-Bench/MLE-Dojo, validating task quality and discriminativeness.

Results¶

606 verified tasks generated from 224 source datasets (avg. 2.71 tasks/dataset).
Average end-to-end preparation time: 419.98 s/task, 1136.20 s/dataset (excluding execution-based verification; per-task execution typically <600 s).
Average generation cost: $0.78/task**, **$2.11/dataset.
LLM performance on a held-out subset of 50 tasks (8 mainstream LLMs) shows strong rank correlation with MLE-Dojo leaderboard results, validating benchmark quality without reporting specific Spearman/Kendall values in the excerpted text.
Modality distribution: Tabular 43.5%, NLP 21.7%, Vision 11.8%, Audio 9.6%, Time Series 9.5%.
Metric distribution: F1/Precision/Recall 24.7%, AUC/ROC 18.3%, RMSE/MAE/MSE 17.3%, custom domain metrics 16.2%.

Limitations¶

Backbone model is GPT-5; generalization of pipeline quality to weaker backbone LLMs is asserted but not fully characterized in the excerpted text.
Source datasets are exclusively from Kaggle; dataset selection bias (filtered by high usability score) may limit coverage of niche or proprietary domains.
Execution-based verification runtime is highly variable and hardware-dependent (GPU/CPU), complicating large-scale deployment cost estimates.
The Brainstormer is capped at 3 candidate formulations per dataset, potentially missing valid task designs.
Non-deterministic Review layer may inconsistently flag semantic issues across runs.

Relevance to Agentic AI / LLM Agents¶

MLE-Smith directly addresses the data scarcity problem for training and evaluating MLE agents—a recognized bottleneck in the agentic AI field—by demonstrating that multi-agent pipelines can themselves generate high-quality agent evaluation environments at scale. The generate–verify–execute paradigm is a reusable architectural pattern for automated benchmark synthesis beyond MLE (analogous to SWE-Smith for software engineering), suggesting a broader principle: agentic systems can self-supply their own training curricula. The hybrid verification design—combining deterministic guards, LLM-mediated semantic review, and execution-in-environment—offers a concrete template for quality assurance in any automated task-generation pipeline. This is particularly relevant to researchers building self-improving agents, synthetic data generators, or scalable evaluation harnesses.