Skip to content

Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery

🕒 Published (v1): 2025-07-09 20:03 UTC · Source: Arxiv · Venue: ICML 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

cmbagent is a ~30-agent LLM system for fully autonomous quantitative scientific research, built on a robotics-inspired Planning & Control (P&C) strategy that requires no human-in-the-loop. It structures every research task as a two-phase pipeline—plan generation/review followed by plan execution—and demonstrates PhD-level cosmology tasks completed end-to-end. On the DS-1000 benchmark, P&C mode outperforms one-shot mode 78% vs. 66% overall success rate.

Problem

Prior multi-agent scientific systems (e.g., Laverick et al. 2024) required human-in-the-loop review at every step, limiting throughput and true autonomy. The gap is full end-to-end automation of quantitative research workflows—download data, write and execute code, interpret results, handle failures—without human intervention.

Method

cmbagent implements a Planning & Control (P&C) strategy layered over the AG2 multi-agent framework:

  • Planning phase: A planner agent generates a structured JSON plan (up to \(n_\text{steps}\) steps, each with a sub-task, action list, and assigned agent). A plan_reviewer agent critiques it for \(n_\text{reviews}\) rounds; formatting and recorder agents produce structured, auditable output.
  • Control phase: A controller distributes plan steps to a researcher (reasoning/interpretation) or engineer (Python coding). Each coding sub-task triggers a nested chat: engineer → formatting agent → executor (local code execution) → post-execution interpreter. On failure, the system retries up to \(n_\text{fails}\) times, with an installer agent for missing packages; on exceeding \(n_\text{fails}\), a terminator agent ends the session.
  • Context management: After each step, all agents and chat history are reset; only a shared "context block" (prior code, execution outputs, researcher summaries) is injected into system messages. This halves token cost relative to accumulating full history.
  • Context agents: Domain-specific agents (e.g., camb, class, classy_sz) are either given full library documentation as a single large context string or use RAG over vector-embedded documentation corpora.

Key Contributions

  • A fully automated, no-human-in-the-loop multi-agent harness for quantitative research tasks, with configurable \(n_\text{steps}\), \(n_\text{reviews}\), \(n_\text{fails}\) hyperparameters.
  • A step-boundary context reset strategy that preserves inter-step memory via a shared context block while reducing session token cost by ~50%.
  • Domain-specific context agents with auto-updating documentation (camb agent regenerates its system message on each library build via sphinx_markdown_builder).
  • Integration as a backend in denario, a larger meta-system that adds idea generation (idea-maker/idea-hater loop), methodology drafting, and automated LaTeX manuscript compilation via LangGraph + Perplexity.
  • Open-source distribution (GitHub, PyPI, Docker, HuggingFace GUI, cloud deployment).

Results

  • camb context agent (gemini-2.5-pro) vs. GPT-4o/GPT-4.1/gemini-2.5-pro on 14 cosmology problems: camb agent outperforms all baselines, particularly on hard problems (12–14) where all three frontier LLMs fail systematically.
  • DS-1000 benchmark (pandas, numpy, matplotlib subsets, gpt-4.1 backend):
  • One Shot: 66% overall success rate
  • Planning & Control: 78% overall success rate
  • Largest gain: pandas (problems 0–10): 0.3 → 0.6; matplotlib: 0.5 → 0.7
  • Cosmology end-to-end task (Union2.1 SNe MCMC parameter estimation for \(H_0\) and \(\Omega_\Lambda\)): solved successfully on first run, autonomously producing MCMC chains, contour plots, and posterior summaries.

Limitations

  • Domain specialization is narrow (cosmology/astrophysics); context agents require manual curation or library-specific build hooks not easily generalized.
  • The P&C strategy requires upfront task specification in a structured form; ill-posed or exploratory tasks may not decompose cleanly into \(n_\text{steps}\) steps.
  • Failure handling is brittle at the boundary: exceeding \(n_\text{fails}\) terminates the entire session rather than gracefully recovering or re-planning.
  • Evaluation is limited to one domain (cosmology) and one general coding benchmark (DS-1000 subset); no cross-domain scientific generalization is demonstrated.
  • The denario integration (full paper-writing pipeline) was not publicly released as of the paper's submission date.
  • No ablation over \(n_\text{steps}\), \(n_\text{reviews}\), or \(n_\text{fails}\) hyperparameters is reported.

Relevance to Harnesses / Meta-Harnesses

cmbagent is a concrete, open-source implementation of a meta-harness pattern: a Planning phase generates a structured execution plan, and a Control phase dispatches sub-tasks to specialized sub-agents—mirroring the plan-then-execute skeleton common to harness designs. The step-boundary context reset with selective state propagation (a "context block" carrying only salient outputs forward) is a directly transferable technique for managing token cost in long-horizon harnesses. The denario integration demonstrates harness composition: cmbagent serves as a pluggable research-execution backend within a higher-order meta-harness that adds idea generation, methodology drafting, and manuscript compilation as additional orchestration layers. The explicit parameterization of \(n_\text{steps}\), \(n_\text{reviews}\), and \(n_\text{fails}\) provides a model for how harness designers can expose control knobs without rewriting orchestration logic.