Skip to content

MaintainCoder: Maintainable Code Generation Under Dynamic Requirements

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MaintainCoder is a multi-agent LLM system that generates structurally maintainable code by embedding the Waterfall model and classical design patterns into a specialized agent pipeline. It introduces MaintainBench, the first benchmark evaluating code maintainability through requirement evolution cycles rather than static test cases. MaintainCoder improves post-modification correctness by over 60% compared to baselines while also improving initial code correctness.

Problem

Existing code generation systems (including multi-agent frameworks like AgentCoder, MapCoder, MetaGPT) optimize for immediate functional correctness but ignore maintainability—the ability to adapt code to evolving requirements with minimal rework. No benchmark quantified maintenance effort under requirement evolution, and traditional static metrics (cyclomatic complexity, Maintainability Index) fail to capture real-world maintenance cost and contradict each other in practice. 60–80% of software lifecycle costs arise from post-deployment maintenance, yet this phase is the least studied.

Method

MaintainCoder orchestrates four specialized LLM agents via AutoGen in two modules:

Code Framework Module: 1. Requirements Analysis Agent — decomposes problem via chain-of-thought, extracts core functions and constraints. 2. Design Pattern Selection Agent — selects classical design patterns (e.g., Strategy, Factory) per functional module to enforce low coupling and high cohesion. 3. Framework Design Agent — constructs class skeleton adhering to single-responsibility principle. 4. Framework Evaluation Agent — reviews the framework for coupling/cohesion/reusability and feeds corrections back to the Framework Design Agent in a loop.

Code Generation Module: 5. Code Generation Agent — implements the blueprint into PEP 8/257-compliant Python, inserting a test assertion for iterative debugging. 6. Code Optimization Agent — executes code against test cases, diagnoses failures via chain-of-thought, and iterates until passing.

MaintainBench extends HumanEval, MBPP, APPS, CodeContests, xCodeEval (~500 problems) with four requirement change patterns (functional extensions πext, interface modifications πint, data structure transformations πdst, error handling enhancements πerr). Dynamic metrics measure: post-modification Pass@k, relative/absolute code change volume (Codediff), and AST structural similarity (ASTsim). Maintainability is formalized as discounted expected cumulative modification cost, approximated via Monte Carlo first-order truncation: M̂(C0) ≈ (1/N)∑M(C0→C1).

Key Contributions

  • MaintainBench: First dynamic benchmark assessing code maintainability through requirement evolution; 500+ problems across five difficulty tiers with four systematic change patterns and expert-curated test cases.
  • Dynamic metrics: Pass@k post-modification, Codediff (relative/absolute line change), ASTsim — shown to be mutually consistent, unlike contradictory static metrics.
  • MaintainCoder: Multi-agent pipeline integrating Waterfall model + design patterns; outperforms all baselines on dynamic maintainability while improving initial correctness.
  • Empirical finding: Static metrics (MI, CC) are unreliable — multi-agent systems like AgentCoder/MapCoder achieve high MI but degrade second-turn Pass@k; dynamic metrics expose this failure.
  • Human baseline: Skilled competitive programmers (Codeforces 1700–2300) produce less maintainable code than MaintainCoder under time pressure.

Results

  • MaintainCoder (GPT-4o-mini) on APPS-Dyn: 50.5% Pass@5 (15–30 points above all baselines including GPT-4o and Claude-3.7-Sonnet), ASTsim 0.797 (+28% over second-best), relative code change 29.4%.
  • MaintainCoder (DeepSeek-V3) on CodeContests-Dyn: 36.7% Pass@5 (+60% over second-best), ASTsim 0.785, relative code change 33.0%, CC ≈ 3 vs. baseline 6–15.
  • Initial correctness (Table 3): MaintainCoder (GPT-4o-mini) improves from 46%→57% Pass@5 on xCodeEval (+24% relative); DeepSeek-V3 backbone: 75%→77%.
  • vs. human programmers (CodeContests-Dyn): Humans achieve 23.5% Pass@5, ASTsim 0.541, 112.3% relative code change; MaintainCoder (GPT-4o-mini) reaches 32.6% Pass@5, ASTsim 0.833, 23.2% change.
  • vs. o3-mini: MaintainCoder (o3-mini) improves Pass@5 from 30.3%→36.4%, ASTsim 0.661→0.794.
  • Ablation: Removing code optimization degrades Pass@5 by 37.13% relative on xCodeEval; removing framework evaluation degrades by 25.71%.
  • Token cost: MaintainCoder uses ~33k tokens (CodeContests), comparable to MapCoder (~38k) and o3-mini (~21k), far below MetaGPT/ChatDev (50k+).

Limitations

  • Evaluated only on Python programming tasks; generalization to other languages is unverified.
  • MaintainBench uses first-order Monte Carlo truncation (single-step requirement change) as a proxy for multi-step evolution; long-horizon maintainability remains unprobed.
  • Competition-level coverage is limited (~120 problems); repository-level tasks (e.g., SWE-Bench) are explicitly out of scope.
  • Multi-agent pipeline increases per-query latency and computational cost relative to single-model baselines.
  • Human evaluation is time-constrained (30 min/problem), which may artificially disadvantage human programmers.
  • Static metrics are critiqued but not replaced for scenarios where dynamic probing is infeasible.

Relevance to Agentic AI / LLM Agents

MaintainCoder is a direct instantiation of structured multi-agent collaboration applied to software engineering, demonstrating that decomposing code generation into role-specialized agents (analyst, architect, designer, evaluator, coder, optimizer) with explicit feedback loops produces structurally superior outputs compared to both monolithic LLMs and existing multi-agent systems. The work highlights that naive multi-agent frameworks (AgentCoder, MapCoder) optimizing for single-round task completion can actively harm long-term code quality—a key failure mode for agentic systems deployed in iterative real-world contexts. MaintainBench's dynamic evaluation protocol is directly applicable to other agentic coding benchmarks that currently rely on static correctness, and the formalization of maintainability as discounted cumulative modification cost provides a principled objective for training or prompting future coding agents.