Skip to content

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

🕒 Published (v1): 2026-06-20 14:57 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent NL2Repo framework; repo-level orchestration is the key harness benchmark frontier

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CodeTeam is a multi-agent LLM framework for NL2Repo (natural-language-to-repository) generation that decomposes the task into competing architectural planning, CTO-guided contract normalization, and dependency-aware implementation with QA-driven repair. It improves SketchBLEU by 4.1 (PE) and 2.9 (SFT) absolute points over CodeS, and achieves 34.6% / 42.3% test pass rates on NL2Repo-Bench in PE/SFT settings.

Problem

Generating a complete software repository from a natural-language requirements document requires architectural design, cross-file interface consistency, dependency scheduling, and iterative error repair — challenges that single-pass LLM generation and prior agent systems (ChatDev, MetaGPT, CodeAgent) each address only partially. No prior system simultaneously handles (1) machine-checkable design planning, (2) cross-file interface consistency throughout implementation, and (3) coordinated bounded-context execution with iterative repair.

Method

CodeTeam orchestrates five role types through a seven-step pipeline:

  1. Requirements Preprocessing (Step 0): Rule-based normalization of README-style input — section heading normalization, nested bullet flattening, retention of only actionable code blocks.
  2. Architect Competition (Step 1): \(N=4\) Architect agents independently generate competing Software Design Sketches (SDSs), each specifying technology stack, fixed file tree, per-file responsibilities, public interfaces, dependency edges, and a developer allocation plan. RAG-grounded retrieval of design references is optional.
  3. CTO Selection (Step 2): A single CTO agent performs comparative analysis across SDSs and selects one, normalizing it into a machine-checkable contract governing all downstream steps.
  4. Materialization (Step 3): Git repository initialized; full file hierarchy pre-constructed; exactly \(D\) Developer agents instantiated per the selected developer allocation plan.
  5. Dependency-Aware Implementation (Step 4): Developer agents implement files following a dependency-aware scheduler (dependency layers executed in order); interface updates propagated via lightweight Git commits read by downstream agents (bounded context).
  6. QA-Driven Repair (Steps 5–6): A QA agent tests completed dependency layers or small file batches against the SDS contract; failures are routed back to responsible Developer agents; impacted dependent files are requeued. Loop continues until no new issues, iteration limit, or resource exhaustion.
  7. Output (Step 7): Final Git repository.

Two variants evaluated: PE (prompt-engineering only, Qwen2.5-72B-Instruct) and SFT (same backbone fine-tuned to follow SDS-centric workflow).

Key Contributions

  • SDS as executable contract: Unlike prior sketch/planning artifacts, the SDS explicitly specifies file ownership, public interfaces, and dependency edges and actively constrains every downstream implementation step.
  • Architect competition + CTO normalization: Multiple competing architectural proposals evaluated by a dedicated selection agent, increasing design diversity while maintaining a single normalized contract.
  • Project-specific developer allocation: Developer count \(D\) and file-to-developer ownership are determined per-project by the selected SDS, not fixed globally.
  • Git-based lightweight coordination: Interface change propagation via Git commits under bounded context, enabling multi-agent parallelism without full-context sharing.
  • External execution-based validation: NL2Repo-Bench (104 Python library tasks evaluated with upstream pytest suites) used as secondary validation beyond the sketch-similarity SketchBLEU metric.
  • Comprehensive ablation: RAG grounding (+8.1% relative SketchBLEU), project-specific developer allocation (+9.9% relative), and Git coordination each isolated.

Results

  • SketchEval / SketchBLEU (vs. CodeS baseline):
  • CodeTeam-PE: +4.1 absolute SketchBLEU points over CodeS-PE
  • CodeTeam-SFT: +2.9 absolute SketchBLEU points over CodeS-SFT
  • NL2Repo-Bench (104 tasks, upstream pytest pass rate):
  • CodeTeam-PE: 34.6% average test pass rate (highest among PE methods)
  • CodeTeam-SFT: 42.3% average test pass rate (highest among SFT methods)
  • Ablation contributions (relative SketchBLEU):
  • Project-specific developer allocation: +9.9% relative
  • RAG-grounded planning: +8.1% relative
  • Git-based coordination: smaller but consistent complementary gain

Limitations

  • Evaluated exclusively on Python repositories (SketchEval and NL2Repo-Bench are Python-only); generalizability to other languages untested.
  • Backbone model fixed to Qwen2.5-72B-Instruct; performance on smaller or different model families unknown.
  • \(N=4\) Architects chosen empirically; the trade-off between design diversity and token cost is not formally characterized.
  • QA agent generates lightweight tests; does not run full upstream test suites during generation (only NL2Repo-Bench evaluation does), so in-loop repair signal may be weaker than the final evaluation signal.
  • SFT variant requires labeled training data following the SDS-centric workflow, creating a data dependency not present in PE.
  • No wall-clock or token-cost reporting, making computational overhead of the multi-architect competition unclear relative to simpler baselines.

Relevance to Harnesses / Meta-Harnesses

CodeTeam is a direct example of a workflow-level meta-harness: it does not improve any single model but instead orchestrates a pipeline of competing planners, a selection/normalization agent, dependency-aware schedulers, and a repair loop — closely analogous to the harness pattern of decomposing a complex task into typed agent roles with explicit handoff contracts. The SDS-as-contract mechanism is particularly relevant: it operationalizes the harness design principle of producing a machine-checkable intermediate artifact that governs downstream agents, preventing the common failure mode of cross-agent inconsistency. The RAG-grounded architect competition also illustrates how a meta-harness can inject external knowledge at the planning stage without modifying individual agents, a pattern directly applicable to research digest or analysis harnesses that must reconcile multiple independent information sources before synthesis.