Skip to content

MOSAIC: Multi-agent Orchestration for Task-Intelligent Scientific Coding

🕒 Published (v1): 2025-10-09 20:35 UTC · Source: Arxiv · Venue: NEURIPS · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MOSAIC is a training-free, four-agent LLM framework for scientific code generation that operates without I/O test cases — a hard requirement for scientific benchmarks like SciCode. It orchestrates Self-Reflection, Rationale, Coding, and Debugging agents in a student-teacher paradigm with a Consolidated Context Window (CCW) to suppress hallucination across chained subproblems. With GPT-4o, MOSAIC achieves 20.01% main-problem and 41.69% subproblem solve rates, outperforming all baselines by up to 24%.

Problem

General-purpose multi-agent coding frameworks (MapCoder, CodeSIM, AgentCoder) rely on sample I/O test cases to verify and iterate on generated algorithms. Scientific coding benchmarks provide only function signatures and domain context — no ground-truth inputs/outputs — making those frameworks inapplicable. Additionally, chained subproblems cause context-window bloat, compounding hallucinations and error propagation across dependent scientific computations.

Method

MOSAIC orchestrates four specialized agents on top of any LLM backbone (LangGraph for communication):

  1. Self-Reflection Agent: Given ground-truth rationales from \(\leq 5\%\) of validation data (teacher module), it iteratively verifies intermediate reasoning steps and produces refined pseudocode before any code is written.
  2. Rationale Agent: Uses pseudocode from Self-Reflection as few-shot examples to generate a step-by-step reasoning plan for unseen test problems. Maintains a Consolidated Context Window (CCW) — only prior function signatures plus one-sentence summaries of each subproblem — to bound context growth and reduce hallucination.
  3. Coding Agent: Translates the plan into executable code, aware of both the subproblem and the broader problem context. Can optionally incorporate sample test cases when available.
  4. Debugger Agent: Executes generated code and runs up to \(k\) rounds of syntactic error correction in collaboration with the Coding Agent.

A Bucketing Module routes problems to domain-specific memory stores (physics, chemistry, biology, materials science, mathematics), preventing cross-domain interference. The student-teacher design is inspired by knowledge distillation: the teacher provides structured few-shot rationales; the student agents learn to replicate that reasoning structure.

Key Contributions

  • Training-free, LLM-agnostic multi-agent framework for scientific code generation without I/O test cases.
  • CCW mechanism that compresses chained subproblem history to function signatures + one-line summaries, mitigating context-window hallucination.
  • Domain-partitioned memory that prevents cross-domain interference across five scientific fields.
  • Ablation studies identifying the relative contribution of each agent to overall accuracy.
  • Up to 24% improvement over prompting baselines and 8.5% over the SciCode best prior method on the SciCode benchmark across three LLM backbones.

Results

  • GPT-4o + MOSAIC: 12/65 main problems (20.01%), 113/283 subproblems (41.69%); vs. SciCode baseline 7/65, 94/283; vs. best competitor LATS 4/65, 49/283.
  • Claude Sonnet 4 + MOSAIC: 13/65 main problems, 118/283 subproblems; vs. SciCode baseline 9/65, 109/283.
  • Gemini 2.5 Flash + MOSAIC: 11/65 main problems, 117/283 subproblems; vs. SciCode baseline 7/65, 112/283.
  • On general-purpose benchmarks (MBPP, HumanEval, APPS), MOSAIC achieves comparable (not superior) performance to MapCoder and CodeSIM.
  • Biology is the hardest domain across all backbones; problems with >10 subproblems remain largely unsolved even with CCW.

Limitations

  • Semantic errors (incorrect algorithm logic) are not addressed — the Debugger Agent only corrects syntactic/execution errors; no mechanism exists to validate algorithmic correctness without ground-truth I/O.
  • CCW degrades on very long chains (>10 subproblems): the framework fails to consistently reuse previously introduced function headers.
  • Biology domain is consistently the weakest due to sparse validation exemplars and concept-level (not numerical) error modes.
  • Performance differences across LLM backbones suggest heterogeneous agent assignment could help, but MOSAIC uses homogeneous backbones within a run.
  • Teacher module requires \(\leq 5\%\) ground-truth code from validation data — not fully zero-resource.
  • Prompt sensitivity: minor wording changes propagate errors through the subproblem chain; the paper's mitigation (capitalization) is heuristic.

Relevance to Harnesses / Meta-Harnesses

MOSAIC is a domain-specific orchestration harness: it wraps multiple LLM agents in a deterministic pipeline (Bucketing → Self-Reflection → Rationale+CCW → Coding → Debugging) with domain-partitioned memory, directly instantiating the meta-harness pattern of routing, specialized sub-agents, and context management. The CCW is a novel harness-level mechanism for bounded context propagation across chained tasks — analogous to state-passing in workflow engines — and is relevant to anyone designing harnesses that must thread intermediate results across long agent chains without full-context replay. The student-teacher initialization pattern (few-shot rationale seeding from a small labeled set) is a reusable harness bootstrapping strategy for domains where test-time I/O is unavailable. The ablation methodology — toggling individual agents — provides a template for empirically attributing gains to specific harness components.