MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models

🕒 Published (v1): 2026-06-23 17:00 UTC · Source: Arxiv · link

Why this paper was selected

Lionel Briand (top SE researcher); automated test oracle generation harness for VLA evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MANGO is a multi-agent framework that automatically generates fine-grained test oracles for Vision-Language-Action (VLA) robotic systems from natural-language task descriptions. It decomposes complex instructions into ordered sequences of atomic tasks, each with a grounded symbolic oracle, replacing brittle hand-authored binary oracles. Evaluated on LIBERO 10 and RoboCasa Humanoid Tabletop, it matches symbolic oracles in failure detection while enabling precise failure localization.

Problem

Existing VLA robot testing relies on manually written symbolic oracles that return a single binary pass/fail based on the final environment state. These oracles are expensive to author, domain-specific, non-reusable across benchmarks, and provide zero diagnostic information about which sub-task within a long-horizon execution failed. As VLA tasks grow in complexity and length, this end-state-only evaluation becomes a critical bottleneck.

Method

MANGO operates in three sequential, one-time-amortizable modules:

1. Atomic Task Library Generation: Given an initial set of natural-language task descriptions, a Generator agent (a "World Model" LLM chosen for physical-semantic reasoning) produces a candidate library of parameterized atomic task templates (e.g., Open(X), Pick(X), Place(X,Y)). An Assessment Board of parallel lightweight LLM Assessors evaluates atomicity, non-redundancy, and coverage. A reasoning Judge LLM synthesizes assessor verdicts and issues structured Refinement Instructions back to the Generator; the loop runs up to 10 iterations.

2. Atomic Task Oracle Generation: For each atomic task in the library, MANGO grounds simulator API functions (contact detection, spatial predicates) to produce parameterized symbolic oracles of the form \(O_{a(\theta)}: s \to \{\text{Pass}, \text{Fail}\}\). Same Generator–Assessors–Judge loop.

3. Fine-Grained Oracle Generation (per task): Each complex instruction \(I\) is decomposed as \(D(I) = \langle a_0(\theta_0), a_1(\theta_1), \ldots, a_{n-1}(\theta_{n-1}) \rangle\) by looking up the atomic library, instantiating with environment-specific objects, and composing oracles with ordering constraints (\(\parallel\) for parallel, \(\prec\) for strict sequence): $\(O_F(I) = \bigwedge_{i=1}^{n} O_{a_i}\)$ The same iterative refinement loop checks oracle logic, object grounding, function compliance, and execution reliability.

Modules I and II run once per benchmark; Module III scales per-task.

Key Contributions

• Formal definition of fine-grained oracles as compositions of parameterized atomic symbolic oracles with explicit temporal ordering constraints (\(\parallel\), \(\prec\)).
• MANGO: the first multi-agent pipeline for automated fine-grained oracle generation for VLA robots, with a reusable three-stage architecture (library → atomic oracles → per-task fine-grained oracles).
• Generator–Assessors–Judge iterative refinement pattern, where parallel lightweight Assessors cover independent quality dimensions and a reasoning Judge arbitrates conflicts.
• Empirical evaluation on two benchmarks (LIBERO 10, RoboCasa Humanoid Tabletop) including ablation studies on component contributions and minimum initial task set size.
• Public replication package (upon acceptance) including oracle libraries and experimental artifacts.

Results

• MANGO-generated fine-grained oracles detect a similar number of failures as existing hand-authored symbolic oracles on both LIBERO 10 and RoboCasa Humanoid Tabletop.
• Fine-grained oracles correctly localize failures to specific atomic sub-tasks, which symbolic oracles cannot do.
• Fine-grained oracles provide richer diagnostic information about failure modes (e.g., distinguishing "failed to open fridge" vs. "dropped bottle during transport").
• Ablation studies confirm contributions of each module component and show the initial task set can be minimized while preserving oracle quality (specific numbers not fully reproduced in the provided excerpt).
• World Model LLMs outperformed other foundation models as the Generator in preliminary evaluation.

Limitations

• Requires simulator functions (contact detection, spatial predicates) to be available and correctly specified; oracle quality is bounded by simulator API expressiveness.
• The iterative refinement cap (10 iterations) may leave some oracles unresolved if the Judge never accepts.
• Atomic task library and oracle library must be rebuilt for each new benchmark/simulator, though they amortize across tasks within a benchmark.
• Evaluation is simulation-only; transfer to physical robots with imperfect state estimation is unaddressed.
• Quantitative results (precision/recall numbers, exact localization accuracy) are not fully provided in the excerpt.

Relevance to Harnesses / Meta-Harnesses

MANGO is a direct instantiation of a meta-harness pattern: it generates the infrastructure (oracle libraries) that downstream test harnesses consume, rather than testing tasks directly. The Generator–Assessors–Judge iterative refinement loop is architecturally isomorphic to a meta-harness pipeline—a orchestrator (Judge) coordinates parallel evaluators (Assessors) and a generator, looping until quality criteria are met. The three-module design (library generation → oracle generation → per-task instantiation) demonstrates how a meta-harness can amortize expensive artifact creation once and reuse it at scale, a pattern directly applicable to digest/pipeline harnesses that build reusable prompt or tool libraries. The structured feedback loop between Judge and Generator is a concrete, empirically evaluated example of self-refining multi-agent harnesses.