MuMuTestUp: Mutation-based Multi-Agent Test Case Update¶

🕒 Published (v1): 2026-05-19 02:25 UTC · Source: Arxiv · link

Why this paper was selected

MuMuTestUp: mutation-based multi-agent test update for CI/CD code evolution

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MuMuTestUp is a mutation-guided multi-agent framework for automatically updating broken or outdated test cases under CI/CD-driven code evolution. It introduces three specialized agents for mutation analysis, fine-grained coverage analysis, and semantic context retrieval, coordinated by a central Coordinator agent that drives iterative refinement. With GPT-4.1, it achieves 88.94% line coverage, 63.36% branch coverage, and 72.39% mutation score on a new pull-request-level Java benchmark.

Problem¶

Existing LLM-based test case update approaches suffer three compounding limitations: (1) they prioritize test executability over assertion adequacy, producing weak oracles with low mutation scores; (2) they provide only coarse line-coverage signals rather than identifying specific uncovered lines and branches; and (3) they rely on exact-matching symbol retrieval, which breaks when LLMs hallucinate symbol names. Together, these cause updated tests to compile and pass without actually detecting regressions.

Method¶

MuMuTestUp structures test case updating as a four-stage iterative loop driven by six specialized agents:

Input Preprocessing agent filters diff hunks (via Repetition and Test TF-IDF Similarity ranking, excluding ground-truth-dependent Breakage TF-IDF), non-test helper methods, and class variables to reduce prompt noise before any LLM call.
Test Update agent produces an initial updated test using filtered context plus any instructions from the Coordinator; on the first iteration, it performs minimal edits to restore correctness.
Coordinator agent extracts generated test code, runs run_test_coverage and run_test_mutation to collect line/branch coverage and mutation scores, maintains the historically best test via record_best, checks termination conditions, and dispatches the appropriate analysis agents.
Three feedback-generating agents are dispatched conditionally:
Error Analysis agent: extracts compilation/runtime error messages, identifies unknown symbols, locates faulty lines, and annotates them with error context.
Coverage Analysis agent: extracts uncovered lines and branches from coverage reports and annotates methods with per-line/per-branch repair instructions rather than exposing raw coverage signals.
Mutation Analysis agent: identifies surviving mutants via run_test_mutation, annotates each surviving mutant with targeted repair instructions to strengthen or generate missing assertions.
Semantic Retrieval agent: builds a ChromaDB vector store over the repository, generates embeddings, and retrieves contextually relevant symbols via semantic similarity to handle hallucinated or missing symbol queries.

All agent outputs are merged via merge_instructions into a unified instruction set fed back to the Test Update agent for the next iteration.

Key Contributions¶

First systematic integration of mutation analysis into automated test updating, using surviving mutants as indicators of assertion weakness.
Multi-agent architecture separating error analysis, coverage analysis, mutation analysis, and context retrieval into six specialized agents with 18 dedicated tools.
Semantic retrieval via ChromaDB vector similarity replacing exact-match language-server retrieval, improving robustness to hallucinated symbol queries.
Prbench: first pull-request-level test update dataset, 571 samples from 10 open-source Java projects, covering both focal-method-changed and focal-method-unchanged cases across multi-commit PR spans, validated by three-round execution.

Results¶

All figures are against the best baseline (TestUpdater) using GPT-4.1 on Prbench:

Line coverage: 88.94% (+5.33% over TestUpdater)
Branch coverage: 63.36% (+19.93% over TestUpdater)
Mutation score: 72.39% (+16.66% over TestUpdater)
Improvements hold with Deepseek-V3.2 as well (magnitude not reported in the provided text excerpt)
TestUpdater failed to produce compilable tests on both motivating examples; ReAccept produced compilable tests but with removed or weakened assertions

Limitations¶

Evaluation is restricted to Java projects; generalization to other languages is not demonstrated.
Mutation testing (PIT or equivalent) is computationally expensive and adds per-iteration overhead that could slow CI pipelines.
Prbench covers 10 projects; cross-project generalization breadth is limited.
The provided text is truncated; quantitative ablation results and Deepseek-V3.2 breakdowns are not fully visible.
Stopping thresholds for coverage and mutation score are predefined hyperparameters whose sensitivity is not analyzed in the available text.

Relevance to Harnesses / Meta-Harnesses¶

MuMuTestUp is a concrete instance of a specialized multi-agent harness where a Coordinator agent acts as the meta-controller: it dispatches subordinate specialist agents, aggregates their structured outputs, maintains state across iterations (best-test tracking, termination logic), and closes the loop between test execution and LLM refinement. This architecture—orchestrator + conditionally-invoked tool-bearing specialists + iterative feedback accumulation—directly parallels the agent-harness design pattern relevant to this research thread. The merge_instructions mechanism is a lightweight structured aggregation layer analogous to harness prompt-assembly stages in broader agentic pipelines. The paper also demonstrates how domain-specific tooling (18 tools spanning mutation runners, vector DBs, coverage extractors) can be systematically embedded into a harness to replace ad hoc prompt engineering.