Skip to content

AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; first systematic failure-attribution framework for LLM agentic systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multi-agent LLM systems fail at alarming rates (up to 86.7%), but automatically identifying which agent and which step caused the failure is an unsolved problem — even SOTA reasoning models achieve below 10% accuracy. AgenTracer introduces an automated pipeline (counterfactual replay + programmatic fault injection) to annotate failure trajectories at scale, then trains AgenTracer-8B via multi-granular RL to outperform proprietary giants like Gemini-2.5-Pro and Claude-Sonnet-4 on failure attribution while delivering 4.8–14.2% downstream task gains to real systems.

Problem

Failure attribution in multi-agent systems — pinpointing the decisive agent \(i^*\) and step \(t^*\) responsible for task failure — is left to manual inspection of verbose trajectory logs. Existing benchmarks (MAST: 200 samples, Who&When: 127 samples) are too small for training. When SOTA reasoning LLMs (OpenAI-o1, DeepSeek-R1) are applied to this task, accuracy falls below 10%, leaving no scalable automated solution for agentic system debugging, data efficiency, or grounded self-improvement.

Method

Formal objective. Given a failed trajectory \(\tau = (s_0, a_0, s_1, \ldots, s_T)\) and a binary outcome function \(\Omega\), the decisive error is the earliest \((i^*, t^*)\) such that replacing \(a_{t^*}\) with an oracle action \(a'_{t^*}\) flips the outcome to success: $\(C(\tau) = \{(i, t) \mid i = \mu(t),\; \Omega(\tau) = 0,\; \Omega(R(\tau, t, a'_t)) = 1\}, \quad (i^*, t^*) = \arg\min_{(i,t) \in C(\tau)} t\)$

Data construction (TracerTraj-2.5K). Two annotation strategies run over trajectories from 6 multi-agent frameworks (MetaGPT, AutoGen, Smolagents, AgentPrune, AFlow, OWL-Workforce) and 7 benchmarks: - Counterfactual replay on failed trajectories: an analyzer agent (DeepSeek-R1) proposes a minimally invasive corrected action \(a'_t\) at each step; the first \(t\) for which \(\Omega(R(\tau, t, a'_t)) = 1\) is labeled the decisive error. - Programmatic fault injection on successful trajectories: a perturbation operator \(\Pi\) corrupts a chosen action \(a_t \to \tilde{a}_t\); if \(\Omega(R(\tau, t, \tilde{a}_t)) = 0\), the injection point is the ground-truth error by construction. Combining both sources yields \(D_{\text{tracer}} = D^- \cup D^+\) with 2,000+ annotated pairs.

AgenTracer-8B training. Qwen3-8B is fine-tuned on \(D_{\text{tracer}}\) via GRPO with a multi-granular reward: $\(R(\hat{p}_k) = \mathbb{I}_{\text{format}} \cdot \left[\lambda \cdot r_{\text{step}}(\hat{t}_k) + (1-\lambda) \cdot r_{\text{agent}}(\hat{i}_k)\right]\)$ where \(\mathbb{I}_{\text{format}}\) gates on well-formed <think>/<answer> output, \(r_{\text{agent}} = \mathbb{I}(\hat{i}_k = i^*)\) is binary, and \(r_{\text{step}}\) applies a Gaussian proximity kernel: $\(r_{\text{step}}(\hat{t}_k) = \exp\!\left(-\frac{(\hat{t}_k - t^*)^2}{2\sigma^2}\right)\)$ A dynamic clipping schedule \(B_s = \max(0.2 B_0, B_0(1 - s/S_{\text{total}}))\) encourages early exploration and late exploitation.

Key Contributions

  • First fully automated pipeline for constructing annotated multi-agent failure trajectories, combining counterfactual replay and programmatic fault injection.
  • TracerTraj-2.5K: 2,000+ high-fidelity trajectory–error-step pairs spanning 6 frameworks and 7 benchmarks across coding, web-agent, and math reasoning domains.
  • AgenTracer-8B: a lightweight Qwen3-8B failure tracer trained with multi-granular GRPO, capable of both step-level and agent-level attribution.
  • New SOTA on the Who&When benchmark, surpassing DeepSeek-R1 by ~12.21% and Gemini-2.5-Pro by ~18.18% at agent-level accuracy.
  • Demonstrated downstream utility: feeding AgenTracer-8B's attributions to MetaGPT and MaAS yields 4.8–14.2% performance gains via automated debugging.

Results

  • Who&When (handcraft), agent-level w/G: AgenTracer-8B 69.10% vs. Claude-Sonnet-4 56.90%, DeepSeek-R1 56.90%, Gemini-2.5-Pro 51.72%, GPT-4.1 43.10%.
  • Who&When (handcraft), step-level w/G: AgenTracer-8B 20.68% vs. Claude-Sonnet-4 17.24%, DeepSeek-R1 13.29%, Gemini-2.5-Pro 9.72%.
  • Who&When (automated), agent-level w/G: AgenTracer-8B 69.62% vs. DeepSeek-R1 66.67%, Gemini-2.5-Pro 61.11%.
  • Who&When (automated), step-level w/G: AgenTracer-8B 42.86% vs. Claude-Sonnet-4 40.65%, DeepSeek-R1 31.32%, GPT-4.1 29.52%.
  • AgenTracer-8B matches or beats Qwen3-Coder-480B-A35B (a ~60× larger model) on step-level attribution on Who&When (automated).
  • Downstream gains of 4.8–14.2% when attribution feedback is fed to off-the-shelf multi-agent systems.

Limitations

  • Step-level accuracy remains low in absolute terms (~20% on the handcraft set), indicating attribution at fine granularity is still an open problem.
  • Counterfactual replay requires access to a ground-truth solution \(G\) and a capable analyzer agent (DeepSeek-R1), both of which may be unavailable in open-ended real-world deployments.
  • Annotation pipeline incurs \(O(T)\) LLM calls per failed trajectory, making large-scale collection expensive.
  • Coverage is limited to 6 multi-agent frameworks; generalization to proprietary or structurally distinct systems is not empirically validated.
  • Table 2 (TracerTraj subset results) is truncated in the provided text, so held-out in-distribution performance cannot be fully assessed here.
  • The w/o G setting consistently degrades performance, confirming the model relies heavily on ground-truth context that is often unavailable at inference time.

Relevance to Agentic AI / LLM Agents

AgenTracer directly targets a foundational reliability gap in multi-agent systems: without automated failure attribution, debugging and self-improvement loops require prohibitive human effort. By producing a scalable data synthesis pipeline and a small, deployable tracer model, this work provides concrete infrastructure for the self-correcting and self-evolving agentic AI paradigm. The multi-granular RL approach — jointly rewarding agent identity and step proximity with a smooth Gaussian kernel — offers a training recipe applicable to other fine-grained agentic evaluation tasks. The result that an 8B specialist model decisively outperforms 400B+ generalist reasoning models also reinforces the broader trend that task-specific specialization via RL is more efficient than scale for agentic sub-problems.