Skip to content

How Generation Architecture Shapes Code Complexity in Multi-Agent LLM Systems: A Paired Study on HumanEval

🕒 Published (v1): 2026-05-29 19:34 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent code generation increases complexity vs single-shot; controlled paired study

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper provides the first systematic measurement of how multi-agent LLM code-generation architecture — not just prompt phrasing — shapes the structural complexity of produced code. Across six configurations, two GPT-4o-family models, and 164 HumanEval tasks (1,968 paired observations), architectures cluster into a lean group and a heavy group separated by a 50–130% complexity gap, with the heavy cluster yielding no pass@1 advantage.

Problem

Multi-agent LLM code-generation pipelines (analyst, coder, tester, debugger) are evaluated almost exclusively on functional correctness (pass@1). Whether the choice of orchestration architecture also affects structural complexity — which drives maintenance cost, review effort, and defect risk — is unexamined. Prior work established prompt-level effects on code complexity but left the architecture-level question open.

Method

The study treats six configurations as binary presence vectors \((R, T, D) \in \{0,1\}^3\) over three architectural layers: role decomposition \(R\) (Analyst + Coder), testing with bounded iteration \(T\) (Tester, ≤3 static-feedback rounds), and runtime debugging \(D\) (Debugger, block-wise CFG execution trace, ≤4 repair loops). The six configurations are Basic \((0,0,0)\), AC \((1,0,0)\), ACT \((1,1,0)\), Debugger \((0,0,1)\), AC+Debugger \((1,0,1)\), and ACT+Debugger \((1,1,1)\).

All 164 HumanEval tasks are solved under every (model, architecture) pair with two GPT-4o-family models (gpt-4o-2024-08-06 and gpt-4o-mini-2024-07-18, temperature = 0), yielding 1,968 within-task paired observations. Five RADON complexity metrics are computed on the generated completion only: SLOC, cyclomatic complexity (CC), and Halstead Volume (\(V\)), Difficulty (\(D\)), and Effort (\(E\)). A paired non-parametric pipeline is applied: Friedman omnibus test, Wilcoxon signed-rank post-hoc with Holm correction, Kendall's \(W\) and matched-pairs rank-biserial effect sizes — run under both all-completions and passing-only conditions, yielding 20 omnibus tests total (10 primary, 10 secondary).

Key Contributions

  • C1. First systematic, statistically rigorous measurement of structural complexity across six widely-used multi-agent LLM code-generation architectures.
  • C2. A within-task paired empirical pipeline (Friedman + Wilcoxon/Holm + Kendall's \(W\) + rank-biserial) adapted from Della Porta et al.'s prompt-level study, with the independent variable shifted from prompt pattern to generation architecture.
  • C3. Cross-model replication within the GPT-4o family showing the architectural complexity effect is robust across the flagship and its cost-efficient sibling — the tradeoff most budget-conscious deployments face.
  • C4. A clean two-cluster finding with practitioner-actionable guidance: lean architectures (Debugger, AC+Debugger) match or exceed heavier ones on pass@1 while producing markedly simpler code.

Results

  • Two-cluster partition: The six architectures collapse into two internally indistinguishable groups — lean (Basic, Debugger, AC+Debugger) vs. heavy (AC, ACT, ACT+Debugger) — separated by a 50–130% complexity gap across all five RADON metrics.
  • Layer attribution (non-additive):
  • \(R\) (analyst–coder split) inflates complexity.
  • \(D\) (runtime debugger) does not inflate complexity; on the AC background it actively deflates it.
  • \(T\) (tester) re-inflates complexity.
  • No correctness payoff: The heavy cluster's additional complexity yields no pass@1 advantage over the lean cluster.
  • Robustness: The two-cluster partition and layer effects replicate across both GPT-4o-family models and persist in the passing-only subset.
  • Architecture vs. prompt phrasing: Della Porta et al. found prompt patterns shift only LOC-related measures; architecture-level intervention shifts all five complexity metrics.

Limitations

  • Scoped to function-level complexity on HumanEval; class-, module-, and repository-level effects are left to future work.
  • Two models only, both from the same provider/pretraining lineage (GPT-4o family); cross-provider and open-source generalization is not tested.
  • The six configurations leave two of the eight \((R,T,D)\) cells unfilled (T=1, R=0), as testing without role decomposition is not engineering-meaningful in this framework.
  • HumanEval tasks are short, self-contained Python functions; complexity effects on larger, real-world tasks may differ.
  • Paper text is truncated before full statistical tables are reported; exact effect sizes per metric per model pair are not available in the provided excerpt.

Relevance to Harnesses / Meta-Harnesses

This paper is directly a study of harness architecture: it decomposes multi-agent orchestration harnesses into three composable layers \((R, T, D)\) and measures their structural side-effects, showing that harness elaboration is not free — the analyst and tester layers inflate code complexity by 50–130% without correctness benefit, while the execution-grounded debugger layer is complexity-neutral or beneficial. For harness designers, the key finding is that conversational/dialogic layers (planning, critique) carry hidden complexity costs that runtime-grounded layers do not, which informs when to add pipeline stages. The paired empirical methodology — fixing the task and varying the harness configuration — is also a reusable evaluation template for harness ablation studies, and the \((R, T, D)\) taxonomy maps onto widely-used harness systems (MetaGPT, ChatDev, AlphaCodium, SWE-agent), making findings transferable across frameworks.