Skip to content

Evaluating Software Process Models for Multi-Agent Class-Level Code Generation

🕒 Published (v1): 2025-11-12 22:53 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically evaluates Waterfall-style multi-agent LLM workflows for class-level Python code generation on ClassEval, using ablation over four sequential stages (Requirement → Design → Implementation → Testing) across three LLM backends. Multi-agent process structure reorganizes rather than uniformly improves performance: it increases code quality metrics while degrading functional correctness for most models, with Testing being the dominant stage. The complete Waterfall pipeline yields the lowest pass@1 for GPT-4o-mini (−37.8%) and DeepSeek-Chat (−39.8%), while Claude-3.5-Haiku improves (+9.5%).

Problem

Prior multi-agent code generation frameworks (ChatDev, MetaGPT, AgileCoder) operate only at the function level; class-level benchmarks (ClassEval) remain unstudied in structured multi-agent settings. It is unknown whether imposing a rigid software process model on LLM agents improves class-level code correctness and maintainability, or which individual development activities drive any observed effects.

Method

Four specialized LLM agents—Requirement Engineer, Architect, Developer, Tester—are instantiated with a unified role-template prompt adapted from SOEN-101. They execute sequentially in a Waterfall communication loop: each stage consumes prior artifacts, self-refines via cross-agent feedback, and produces a formal document (requirements doc → design doc → code → test report). The Developer receives up to three bug-fix cycles from the Tester's report.

Four configurations are compared via ablation: \(\text{Waterfall}_\text{Full}\), \(\text{Waterfall}_{w/o\,Req}\), \(\text{Waterfall}_{w/o\,Design}\), \(\text{Waterfall}_{w/o\,Testing}\), all against a RawPrompt baseline (direct single-shot generation). Evaluation uses (1) Pass@1 (class-level and function-level) from ClassEval and (2) SonarQube static analysis covering Software Quality (Security, Reliability, Maintainability) and Clean Code Attributes (Consistency, Intentionality, Adaptability, Responsibility). Models tested: GPT-4o-mini, DeepSeek-V3, Claude-3.5-Haiku; temperature 0.8; ClassEval's 100 holistic class-generation tasks.

Key Contributions

  • First systematic study applying a Waterfall multi-agent workflow to class-level code generation (ClassEval).
  • Ablation design isolating per-stage contribution to correctness and code quality across three LLM backends.
  • Finding that process structure shifts failure modes: structural errors (missing code) decrease while semantic failures and return mismatches increase.
  • Identification of Testing as the dominant stage: improves verification coverage but introduces new reasoning and validation errors; removing it yields highest code quality scores but lowest correctness.
  • Public release of code and experimental data.

Results

  • GPT-4o-mini full Waterfall: class-level Pass@1 −37.8% vs. RawPrompt; runtime errors increase 10–53%; SonarQube maintainability improves.
  • DeepSeek-Chat full Waterfall: Pass@1 −39.8%; similar error increase profile.
  • Claude-3.5-Haiku full Waterfall: Pass@1 +9.5%, the only model to benefit.
  • Removing Testing (\(\text{Waterfall}_{w/o\,Testing}\)) yields the worst correctness but highest code quality scores across models.
  • Removing Requirement or Design stages has comparatively minimal effect on Pass@1 or error type distribution.
  • Most frequent new runtime errors introduced by Waterfall: ValueError, AssertionError, TypeError.
  • Structural failures (missing code) decrease under Waterfall; Semantic Failures and Return Mismatches increase.

Limitations

  • Single process model evaluated (Waterfall); Agile/Scrum/TDD variants not compared within this study despite SOEN-101 suggesting they behave differently.
  • Only three lightweight LLM backends tested; results may not generalize to larger frontier models.
  • ClassEval is Python-only and 100 tasks; cross-language and larger-scale generalization is unconfirmed.
  • Self-refinement loops are capped (three bug-fix cycles); optimal iteration budgets are not studied.
  • SonarQube static analysis does not capture runtime semantic correctness beyond what Pass@1 already measures.
  • The paper text provided is truncated; full quantitative table (Table 2) is absent, limiting verification of per-configuration numbers.

Relevance to Harnesses / Meta-Harnesses

This work is a direct empirical study of a sequential multi-agent harness: the Waterfall workflow is itself a harness that chains specialized sub-agents with formal artifact handoffs and self-refinement loops, exactly the structure that meta-harness designers implement. The ablation methodology—removing one pipeline stage at a time—is a principled harness evaluation technique that produces stage-level attribution of quality and correctness effects, directly informing how meta-harness designers should sequence or weight stages. The finding that over-structuring a harness (full Waterfall) degrades task accuracy while under-structuring it (RawPrompt) sacrifices code quality is a concrete trade-off that any harness architect must navigate. The model-dependence of outcomes (Claude benefits, others regress) warns that harness topology cannot be evaluated independently of the backend model.