Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modeling

🕒 Published (v1): 2025-11-05 12:12 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ProjectGen is a three-stage multi-agent framework for project-level code generation (architecture design → skeleton generation → code filling) that uses a Semantic Software Architecture Tree (SSAT) as a structured intermediate representation to bridge natural-language requirements and executable code. The authors also release CodeProjectEval, a dataset of 18 real-world Python repositories averaging 12.7 files and 2,388.6 LOC with executable test cases. ProjectGen achieves 52/124 test cases on DevBench (57% improvement) and 310 test cases on CodeProjectEval (~10× over baselines).

Problem

Project-level code generation is under-served by three compounding failures: (1) existing datasets are unrealistically small (≤5 files, ≤500 LOC) and use similarity-based metrics (e.g., SketchBLEU) that assign high scores to non-compiling code; (2) human-written requirement documents contain multi-modal elements (diagrams, informal prose) that LLMs cannot reliably parse, creating a semantic gap to executable code; (3) hierarchical cross-file dependencies and LLM context-length limits cause early-stage errors to cascade through dependent modules without intermediate correction.

Method

Dataset — CodeProjectEval: 18 Python repositories (500–10,000 LOC, ≥3 files, ≥80% unit-test coverage) sourced from Commit0, DevEval, and CoderEval, augmented with PRD-style documentation. Each task includes 186 unit tests on average at 90.7% line coverage, enabling executable automatic evaluation.

Framework — ProjectGen (three stages): 1. Architecture Design: An LLM agent converts the PRD into a Semantic Software Architecture Tree (SSAT) — a tree-structured, machine-parsable JSON/XML artifact that encodes modules, files, classes, and functions together with their natural-language responsibilities and inter-component dependency edges. The SSAT is the authoritative blueprint for all downstream agents. 2. Skeleton Generation: A second agent instantiates the SSAT into a compilable scaffold where every function body is initially pass, establishing import structure and call signatures before any implementation is written. 3. Code Filling: Per-file agents fill each function stub in the skeleton, using the SSAT for dependency context. A memory-based context management mechanism retains only the most relevant prior file contents rather than the entire growing codebase, staying within LLM context limits.

Across all stages, a judge agent evaluates intermediate outputs and triggers iterative refinement loops before proceeding to the next stage, catching errors early rather than post-hoc.

Key Contributions

• CodeProjectEval: first project-level dataset with both realistic scale (avg. 2,388.6 LOC, 12.7 files) and executable test-case evaluation.
• SSAT: a structured semantic intermediate representation that makes requirements machine-parsable and provides a shared coordination artifact across agents.
• ProjectGen: three-stage multi-agent pipeline with intra-stage judge-feedback refinement loops and memory-based context management.
• State-of-the-art results on both DevBench and CodeProjectEval.

Results

• DevBench: 52/124 test cases passed — a 57% improvement over the best prior baseline.
• CodeProjectEval: 310 test cases passed — approximately 10× higher than baselines.
• Baselines compared against include CodeS/SketchEval-style one-shot pipelines and MetaGPT/ChatDev-style multi-agent systems (exact per-baseline numbers not visible in the provided excerpt).

Limitations

• CodeProjectEval contains only 18 repositories (Python only), limiting statistical power and language generalizability.
• Repository size is capped at 10,000 LOC by design; behavior on larger industrial codebases is untested.
• Iterative refinement via a judge agent adds significant inference cost; latency/cost trade-offs are not quantified in the provided text.
• SSAT construction itself can be erroneous; error propagation from a malformed SSAT is not ablated in the visible sections.

Relevance to Harnesses / Meta-Harnesses

ProjectGen is a concrete instantiation of a staged, feedback-driven code-generation meta-harness: it decomposes a monolithic task into sequenced sub-agents, each with a defined input/output contract (PRD → SSAT → skeleton → filled code), exactly the pattern studied in harness design. The SSAT functions as the harness's shared state object — a structured intermediate that all agents read and write against, analogous to a scratchpad or task graph in more general meta-harness frameworks. The judge-feedback refinement loop embedded at each stage mirrors the "verify-before-advance" guard pattern common in robust multi-agent harnesses. For researchers tracking this topic, the key transferable insight is using a domain-specific structured representation (SSAT) as the inter-agent coordination primitive rather than raw text or conversation history.