Where Did It Go Wrong? Capability-Oriented Failure Attribution for Vision-and-Language Navigation Agents¶

🕒 Published (v1): 2026-04-28 03:08 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Capability-oriented failure attribution for VLN agents diagnoses perception and planning gaps

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CanTest is a capability-oriented testing framework for Vision-Language Navigation (VLN) agents that attributes task failures to specific agent sub-capabilities (perception, memory, planning, decision) rather than treating failure at the system level. It combines adaptive fuzzing via LLM-driven instruction mutation, per-capability oracles, and counterfactual-reasoning-based failure attribution. Across three state-of-the-art VLN models, CanTest discovers 23–34% more failure cases than the best prior baseline while producing fine-grained, capability-level diagnoses.

Problem¶

Existing testing methods for embodied agents treat the agent as a monolithic system, flagging task-level failures without identifying which sub-capability caused them. Because capabilities (perception, memory, planning, decision) are tightly interdependent, errors cascade and compound over long trajectories, making root-cause localization intractable with system-level metrics alone. There is no prior automated framework that generates test cases explicitly targeting individual capabilities and attributing failures to a specific capability error.

Method¶

CanTest operates in three integrated modules:

1. Adaptive Test Case Generation. LLM-generated task instructions are seeded from 3D scene assets (HM3D dataset, with room and object annotations $A_r, A_o$). A seed corpus is maintained; each seed has a feedback score. Seeds are sampled with probability proportional to their score: $$p_{cs_i} = \frac{\max(F_{cs_i}, 0)}{\sum_{i=1}^{N} F_{cs_i}}$$ Two mutation operators—mild (small semantic change, preserves failure-relevant content) and aggressive (large change, diversifies trajectories)—are applied adaptively based on current feedback.

2. Capability Oracles Construction. Four oracles provide independent, per-step evaluations: - Perception ($\epsilon^p_t$): weighted IoU between agent bounding boxes and RAM image-model detections, combined with LLM-judged annotation similarity (Eq. 3). - Memory ($\epsilon^m_t$): LLM semantic similarity between current agent memory and expected historical visual annotations $VA^{gt}_{1,\ldots,t-1}$ (Eq. 4). - Planning ($\epsilon^{pl}_t$): $1 - \text{nDTW}(\tau^{pl}_t, \tau^{gt}_{t,\ldots,n})$, comparing the agent's planned route to a greedy-expert route (Eq. 5). - Decision ($\epsilon^d_t$): binary mismatch between chosen action $D_t$ and planned action $D^{pl}_t$ (Eq. 6).

3. Capability-Oriented Failure Attribution. Counterfactual intervention: for each detected capability error at time $t$, the agent output is replaced with the oracle's expected output, and the trajectory is re-rolled. If correction turns a failure into success, the error is failure-inducing; among multiple such errors, the earliest is designated the failure-source. The composite feedback score is: $$F_{cs} = F^f + \lambda_{C_x} F^c$$ where $F^f \in \{0,1\}$ is failure-oriented feedback, $F^c = \text{Norm}(\epsilon^{x}_{t^*})$ is capability-oriented feedback from the source error, and $\lambda_{C_x} = N^{C_y}/N^{C_x}$ adaptively down-weights capabilities already well-covered in the failure set.

Key Contributions¶

First automated test case generation framework targeting individual sub-capabilities of embodied agents.
Four automatically constructed capability oracles (perception, memory, planning, decision) with independent evaluation metrics for VLN agents.
Counterfactual-reasoning feedback mechanism for attributing a task failure to a unique failure-source capability and earliest failure-source time step.
Composite feedback score combining failure-oriented and capability-oriented signals to guide adaptive, coverage-balanced test generation.

Results¶

Total failure cases discovered (vs. best baseline VLATest): +23.34% to +33.70% across three target VLN models (ApexNav, MGDM, Mem2Ego) over a 4-hour testing window.
Per-capability failure counts (Table 1, CanTest vs. VLATest+OA, selected highlights):
Perception: 72.2 / 74.7 / 61.4 vs. 51.8 / 56.4 / 50.1 (ApexNav/MGDM/Mem2Ego)
Memory: 66.3 / 56.1 / 42.8 vs. 54.1 / 50.4 / 26.9
Planning: 52.5 / 49.3 / 66.1 vs. 45.1 / 41.1 / 39.7
Decision: 59.5 / 64.7 / 63.4 vs. 52.1 / 50.4 / 58.1
Oracle fidelity (failure case repair rate using capability oracles): 81.30%–96.69%, validating oracle accuracy.
Ablation confirms both failure-oriented and capability-oriented feedback components each contribute independently to discovering more failures.

Limitations¶

Evaluated only on VLN tasks in simulation (Habitat3/HM3D); generalizability to other embodied agent types or real-world deployment is unproven.
Capability oracle construction relies on privileged expert-model access (global map, greedy pathfinder, RAM image annotator) not always available outside simulation.
LLM-in-the-loop for annotation similarity scoring introduces cost and potential inconsistency.
Attribution assumes a single failure-source capability (unique $C_x^*$); concurrent multi-capability failures are not modeled.
Comparison baselines are system-level testers; no capability-oriented baseline exists, so the OA plug-in variant is a proxy rather than a true independent competitor.

Relevance to Agentic AI / LLM Agents¶

VLN agents are a canonical instance of multi-capability LLM-grounded agents where language understanding, spatial memory, planning, and action selection must compose correctly—exactly the failure modes that concern practitioners building autonomous LLM agents. CanTest's counterfactual attribution methodology is directly applicable beyond navigation: any agent with separable sub-modules (retrieval, reasoning, tool use, output) could be instrumented with analogous oracles to pinpoint which module causes downstream task failure. The adaptive fuzzing approach using LLM-generated test cases is also a reusable paradigm for stress-testing instruction-following agents at capability granularity, filling a diagnostic gap that benchmark leaderboards cannot address.