Skip to content

Multi-Agent LLM-based Metamorphic Testing for REST APIs

🕒 Published (v1): 2026-05-27 11:24 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent metamorphic testing for REST APIs; LLMs discover metamorphic relations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ARMeta is a multi-agent LLM workflow that automates metamorphic testing of REST APIs by deriving metamorphic relations (MRs) from OpenAPI specifications, expressing them as Given-When-Then scenarios, and synthesizing executable Python/Behave test code. It addresses the test oracle problem in REST API testing, where correct outputs are often undefined, by checking relational properties between seed and follow-up API executions. Evaluation on two benchmark services (PetStore, UserManagement) shows ARMeta generates complementary coverage relative to scenario-based baselines.

Problem

REST API testing suffers from the test oracle problem: OpenAPI specifications rarely define expected outputs, so automated tools default to shallow checks (HTTP status codes, schema conformance). Metamorphic testing (MT) can sidestep this by checking relations between multiple executions rather than fixed expected values, but no prior work provides an end-to-end, specification-driven pipeline that (1) infers MRs directly from OpenAPI documents, (2) grounds them as seed–follow-up test scenarios, and (3) synthesizes and executes them while handling LLM instability and spec–implementation mismatches.

Method

ARMeta is a three-layer iterative pipeline controlled by a Test Manager:

  1. Generation layer (CrewAI): An MR Generator Agent proposes a bounded batch of high-level metamorphic tests (HLMTs) in Given-When-Then (GWT) format, parameterized by the OpenAPI spec and the accumulated history of prior HLMTs to prevent duplicates. Output is normalized into a JSON list via a lightweight tolerant parser.

  2. Refinement layer (direct LLM API calls, stronger model): An MR Refiner Agent filters HLMTs whose claimed relations cannot be supported by the spec. A Code Refiner Agent validates generated Behave step code via Python ast.parse(), triggering a bounded repair loop (≤2 attempts) on syntax failures; unfixable cases produce no-op placeholder steps.

  3. Execution layer (Behave framework): The assembled test suite is run against the system under test (SUT). Pass/fail outcomes, error traces, and operational coverage (% of unique endpoint–method pairs exercised) are collected. If stopping criteria (coverage target, plateau window, request budget, or 30-minute time budget) are unmet, a new iteration begins.

Different LLM models (GPT-4o for generation, a stronger model for refinement) are assigned per layer. A Streamlit GUI exposes configuration and results.

Key Contributions

  • End-to-end specification-driven MT workflow: OpenAPI → HLMTs → EMTs → execution report, without manual intervention.
  • GWT-structured HLMT format enables both human inspection and reliable LLM processing of metamorphic scenarios.
  • Iterative, coverage-steered test generation with explicit stopping criteria (coverage target, plateau, budgets).
  • Two-stage LLM separation: stable model generates, stronger model refines — improving repeatability under LLM non-determinism.
  • AST-based syntax validation with bounded repair loop and graceful placeholder fallback.
  • Operational coverage tracking to guide generation and audit which documented API operations are exercised.

Results

  • Evaluated on PetStore (19 operations) and UserManagement (28 operations, Swagger 2.0, RBAC).
  • ARMeta explores API behaviors that complement LogiAgent (the scenario-based multi-agent baseline); the two approaches exercise overlapping but non-identical sets of operations and failure modes.
  • Specific pass/fail counts and coverage percentages are not fully reproduced in the provided excerpt (evaluation section is truncated after metric definitions).
  • Execution outcomes are split into assertion failures (MR violated), scenario failures (HTTP 5xx, undocumented status codes, runtime errors before reaching the MR assertion), and passes.

Limitations

  • Full determinism is unachievable in LLM-based pipelines; repeatability is improved but not guaranteed.
  • Operational coverage is approximate and not used as a head-to-head comparison score.
  • The single illustrative JSON example in the generation prompt may mildly bias surface wording of generated HLMTs toward the example domain.
  • Evaluation is limited to two small benchmark services; generalizability to large or complex APIs is unconfirmed.
  • Placeholder steps are inserted when code repair fails, silently reducing effective test coverage.
  • RBAC-protected endpoints (UserManagement) require authenticated sessions; handling of access-control context in MT is not fully detailed.
  • Follow-up executions can fail due to spec–implementation mismatches before any MR assertion is reachable, producing scenario failures rather than true MR violations.

Relevance to Harnesses / Meta-Harnesses

ARMeta is a concrete example of a multi-agent meta-harness: a pipeline controller (Test Manager) orchestrates specialized sub-agents (MR Generator, MR Refiner, Test Generator, Code Refiner) across layered generation, refinement, and execution stages, with iterative feedback loops driven by coverage metrics and stopping criteria. The architectural pattern — capability-differentiated agents, bounded repair sub-loops, artifact normalization between stages, and session-level aggregation — directly parallels the concerns of research harness design. The explicit treatment of LLM instability via model-tier separation and AST validation is a transferable technique for any harness that synthesizes executable artifacts from LLM outputs.