Skip to content

IntentTester: Intent-Driven Multi-agent Framework for Cross-Library Test Migration

🕒 Published (v1): 2026-06-24 08:59 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent test migration harness; concrete scaffold reuse pattern across library boundaries

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IntentTester is a five-agent framework that migrates unit tests across libraries and programming languages by abstracting tests into a language-agnostic Test Description Language (TDL) rather than mapping API signatures. A repository graph of the target codebase is queried semantically to retrieve complete dependency context, and an iterative verification loop produces executable tests. On nine open-source projects it achieves 85% syntactic correctness and 74% execution success, far above structure-mapping baselines.

Problem

Existing cross-library test migration tools (MUT, METALLICUS) rely on structural API-signature mappings, which fail when libraries diverge in design, and fail entirely across languages. They produce non-executable migrated tests that require manual adaptation, blocking automated test-knowledge reuse across libraries with overlapping functionality.

Method

Preprocessing: ANTLR parses the target repository into an AST; classes, methods, fields, and tests are extracted as nodes in a Neo4j repository graph with 11 typed edge relations (INHERITS, CALLS_METHOD, TESTS_METHOD, etc.). Each node is annotated with an LLM-generated natural-language summary.

Five-agent pipeline coordinated by an Orchestrator:

  1. Intent Abstractor Agent — splits multi-purpose source tests into single-intent sub-tests, then converts each into a structured TDL encoding Metadata, Inputs, ExecutionSteps, and Assertions in language-agnostic natural language.
  2. Intent Alignment Agent — embeds TDL steps and performs semantic retrieval over the repository graph, expanding multi-hop dependencies to assemble a complete context bundle (\(C\)).
  3. Planning Agent — judges whether \(C\) is sufficient to implement the TDL; triggers one refinement pass or rejects the test (\(\emptyset\)) if still insufficient.
  4. Test Migration Agent — synthesizes the target-language test by grounding TDL intent in the context bundle and reference patterns from the graph.
  5. Verification Agent — validates syntactic and semantic consistency; on failure feeds corrective context back to the Orchestrator, which retries the full pipeline.

Key Contributions

  • Introduces intent-driven test migration as a paradigm, replacing brittle structural API mappings with semantic TDL abstraction.
  • Five-agent orchestration (Abstractor → Alignment → Planning → Migration → Verification) with iterative feedback loops enabling fully automated, manual-adaptation-free migration.
  • ANTLR-based repository graph with 11 edge types stored in Neo4j, supporting multi-hop dependency retrieval across both Java and Python.
  • Benchmark of 9 real open-source projects (JSON, HTML, Time domains; Java and Python); 2,058 source tests → 3,257 filtered sub-tests → 2,776 syntactically correct migrated tests.
  • Discovery and responsible disclosure of 25 real defects (stack overflows, null dereferences, parsing inconsistencies) confirmed or patched by library maintainers.

Results

  • Syntactic correctness: IntentTester 85% vs. MUT 51% vs. METALLICUS 43% (+30–40 pp).
  • Execution success (effectiveness): 2,410 / 2,776 correct tests execute successfully → 74% effectiveness rate.
  • Defects found: 25 previously unknown defects across JSON, HTML, and Time libraries; several acknowledged/patched by maintainers.
  • Evaluated on 9 open-source projects across 3 domains and 2 languages (Java, Python).

Limitations

  • Pipeline performs one refinement retry in the Planning stage; pathological dependency gaps still result in rejection (\(\emptyset\)) rather than a deeper search.
  • ANTLR-based extraction currently supports only Java and Python; extending to other ecosystems requires new grammar/traversal work.
  • LLM-generated node summaries may be incorrect for poorly documented or highly idiomatic code, silently degrading alignment quality.
  • The 26% of syntactically correct tests that fail execution are not further diagnosed or repaired automatically.
  • Evaluation is limited to three functional domains (JSON, HTML, Time); generalizability to other domains is unverified.

Relevance to Harnesses / Meta-Harnesses

IntentTester is a direct instantiation of a multi-agent harness: a fixed orchestration layer decomposes a complex software-engineering task into specialized sub-agents, enforces a sequential pipeline with conditional retry loops, and gates progress on explicit sufficiency checks — exactly the architecture pattern studied in harness and meta-harness research. The TDL acts as the shared intermediate representation (akin to a plan or task spec) passed between agents, while the Verification Agent's feedback loop mirrors the self-repair loops seen in agentic code-generation harnesses. For researchers tracking this space, IntentTester provides a concrete, evaluated instantiation of how to decompose a structured SE task (test migration) into an agent pipeline with graph-RAG retrieval, planning gates, and iterative verification — a reusable template for other cross-repository reuse or transformation tasks.