CoMind: Towards Community-Driven Agents for Machine Learning Engineering¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Ameet Talwalkar (CMU); agents leveraging community knowledge for ML engineering automation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CoMind is a community-driven multi-agent system for automated ML engineering that simulates the collaborative knowledge-sharing dynamics of Kaggle competitions. It introduces MLE-Live, a live evaluation framework supplying time-stamped community discussions and code kernels, and achieves a 36% medal rate on 75 past Kaggle competitions—state of the art—while outperforming 92.6% of human competitors across eight live competitions.

Problem¶

Existing LLM-based ML engineering agents (AIDE, R&D-Agent, ML-Master, etc.) operate in isolation, ignoring the community knowledge-sharing that characterizes real competitive ML platforms like Kaggle. This causes them to plateau by converging on repetitive strategies. Simultaneously, existing benchmarks (MLE-Bench, DSBench) evaluate agents in static, closed-world settings that omit collaborative dynamics, providing no mechanism to measure community-knowledge utilization.

Method¶

MLE-Live extends MLE-Bench by providing agents with a time-stamped stream of 12,951 discussions and 15,733 Kaggle kernels (published before competition deadlines) alongside vote counts, public scores, author tiers, and referenced datasets—mirroring the information constraints of human participants.

CoMind is a five-role multi-agent system operating in an iterative loop:

Coordinator: Samples high-potential kernels \(K_t'\) and datasets \(D_t'\) from the community state \((K_t, D_t, T_t)\), translates ideas into solution drafts \(S_t\), and publishes successful results back to the community.
Analyzer: Scores each community artifact on novelty, feasibility, effectiveness, and efficiency (0–10) and produces structured reports \(R_t\).
Idea Proposer: Uses \(R_t\) and a persistent memory \(I_t^*\) to brainstorm, filter, and accumulate ideas (\(I_{t+1}^* = I_t^* \cup I_t\)).
Coding Agents (parallel, \(n=4\)): Implement solution drafts via ReAct-style iterative coding/debugging in persistent Jupyter sessions; a monitor LLM terminates failed runs early.
Evaluator: Constructs a held-out validation split \((V_x, V_y)\) invisible to Coding Agents, scores submissions using the official competition metric \(\phi(\hat{V}_y, V_y)\), and maintains a global leaderboard.

The community is represented as a dependency graph \(T_0 = (V, E)\) initialized from top-\(k\) kernels and discussions; each iteration expands this graph with newly generated artifacts.

Key Contributions¶

MLE-Live: A reproducible, temporally grounded evaluation framework simulating realistic Kaggle communities with time-stamped kernels, discussions, and referenced artifacts for 75 past competitions.
CoMind: A community-augmented multi-agent system with five specialized roles achieving state-of-the-art medal rates on MLE-Bench.
Iterative parallel exploration: Mechanism that simultaneously runs multiple Coding Agents per iteration while continuously feeding successful results back into the community pool, balancing exploration breadth with implementation depth.
Live deployment validation on eight ongoing Kaggle competitions with official leaderboard submissions.

Results¶

MLE-Bench (75 competitions, Any Medal rate): - CoMind (o4-mini): 36.0% overall (Low: 59.1%, Medium: 23.7%, High: 33.3%) - Neo multi-agent (prior SOTA, closed-source): 34.2% - R&D-Agent (o3 + GPT-4.1): 30.2% - ML-Master (deepseek-r1): 29.3% - AIDE (o1-preview): 16.9%

Live Kaggle competitions (8 competitions, official leaderboard): - Average rank: top 7.35% (outperforms 92.6% of human competitors on average) - Top 5% on 3 leaderboards; top 1% (0.2% rank) on Playground S5E9 - Improved upon the best public kernel on 5 of 8 competitions

Ablation (20 MLE-Bench-Lite competitions, win rate): - CoMind: 0.668 vs. AIDE+RAG: 0.510, AIDE+Code: 0.469, AIDE: 0.512 - CoMind w/o community resources causes significant drops across all metrics

Limitations¶

Uses a closed-source, proprietary backbone (o4-mini-2025-04-16), limiting reproducibility for researchers without API access.
Hardware budget (single A6000 GPU, 24h per competition) may not reflect resource-constrained deployment scenarios.
Public dataset access was restricted to code-only (no referenced datasets) to reduce contamination, potentially underutilizing available community artifacts.
Evaluated on Kaggle-style competitions only; generalization to other ML engineering contexts (e.g., production deployment, novel research benchmarks) is not demonstrated.
The simulated community is initialized from historical Kaggle data; the framework does not model adversarial or low-quality community contributions.

Relevance to Agentic AI / LLM Agents¶

CoMind directly advances the frontier of LLM agents operating in open-ended, competitive, real-world environments by introducing community knowledge as a first-class input rather than treating external information as a static retrieval source. The iterative parallel exploration mechanism—where multiple agents simultaneously develop solutions and feed results back into a shared knowledge graph—is a concrete architecture for scaling multi-agent collaboration beyond simple task decomposition. The live competition deployment (with official leaderboard validation) provides one of the strongest real-world agentic capability demonstrations to date, avoiding the contamination risks that plague static benchmark evaluations. For researchers tracking agentic AI, this work establishes a new paradigm: agents should not merely use tools but actively participate in and contribute to collaborative knowledge ecosystems.