Skip to content

AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents

🕒 Published (v1): 2026-07-21 06:21 UTC · Source: Arxiv · link

Why this paper was selected

Open-source toolkit for LLM agent failure attribution and root-cause recovery

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentDebugX is an open-source LLM agent debugging framework that closes the loop between failure detection, root-cause attribution, repair proposal, and verified rerun. Its core component, DeepDebug, is a multi-turn diagnostic agent that outperforms single-pass attribution methods on the Who&When benchmark (28.8% vs 21.7% strict accuracy) and repairs 2–3× more GAIA failures than decoupled self-correction baselines.

Problem

LLM agent failures are causally displaced: the step where an error surfaces is often not the step that caused it. Existing observability tools (LangSmith, Langfuse) replay traces but leave root-cause localization to developers; failure taxonomies and attribution benchmarks exist as standalone analyses rather than deployable infrastructure; self-correction methods lack localization grounding and thus underperform. No prior system unifies detection, attribution, recovery, and verified rerun into a single closed loop with a shared trace format.

Method

AgentDebugX structures debugging as a four-stage loop—Detect → Attribute → Recover → Rerun—operating on a portable AgentTrajectory representation that normalizes framework-specific events (LangGraph, CrewAI, OpenAI Agents SDK, OpenTelemetry, ReAct) into an ordered sequence of AgentEvent records.

Detection first applies deterministic rule packs (no LLM call) for mechanically verifiable failures (malformed tool calls, no-progress loops), then falls back to an LLM judge that returns typed findings against a 19-mode seed taxonomy spanning planning, memory, tool use, verification, and coordination. The taxonomy is extensible: a residual inducer clusters novel failures by embedding similarity and proposes new modes for human review.

Attribution escalates through cost-ordered strategies—heuristics, single-pass whole-trace reading, binary search, per-step inspection, budgeted ensembles—before reaching DeepDebug, a multi-turn read-only diagnostic agent: 1. Global read: full trace scan, initial candidate step named. 2. Structure-guided probe: for multi-agent traces, walks the handoff cascade upstream; for single-agent traces, bisects the step range to yield an independent second candidate. 3. Cross-examination: if the two passes disagree, both candidates are inspected side-by-side and the stronger causal explanation is selected, reducing root-cause selection to adjudication between two hypotheses. 4. Diagnosis and suggestion: emits a structured report with responsible agent/step, plain-language explanation, quoted evidence, and one concrete fix.

Recovery packages the diagnosis into a retry directive behind an explicit human or policy gate. Rerun applies the fix from a checkpoint, scores the new branch against the objective, and re-enters detection on failure. An opt-in Error Hub stores scrubbed trajectory–diagnosis–repair bundles as shared debugging memory and CI regression fixtures.

Key Contributions

  • Closed-loop Detect → Attribute → Recover → Rerun debugging framework with a framework-agnostic portable trace format.
  • DeepDebug: a multi-turn, read-only root-cause diagnostic agent using two-reading adjudication (global + structure-guided), outperforming single-pass strategies on long traces (>40 events).
  • Extensible 19-mode failure taxonomy with an automated inducer for novel failure-mode discovery.
  • Policy-gated recovery that routes DeepDebug's localized fix directly to rerun, avoiding an extra model call.
  • Error Hub for cross-team sharing of scrubbed incident bundles as reusable debugging memory and CI fixtures.
  • Multi-surface deployment: Python library, CLI, no-build web console, installable agentic skill (compatible with Claude Code, OpenClaw, Hermes).

Results

  • Who&When attribution (n=184, qwen3.5-9b, diagnosed by gemini-2.5-flash):
  • DeepDebug strict agent-and-exact-step accuracy: 28.8% vs. 21.7% for the strongest single-pass baseline (All-at-Once), 17.9% for Step-by-Step, 17.4% for Binary-Search.
  • Responsible agent accuracy: 56.0% vs. 47.8% (All-at-Once).
  • Agent + step ±1: 32.1% vs. 23.9%.
  • On qwen3.6-27b: DeepDebug 38.0% vs. 36.4% strict (smaller margin; hosted backbone already strong).
  • GAIA end-to-end recovery (165 tasks; 73 initially failed by vanilla qwen3.5-9b):
  • DeepDebug repairs 13/73 failed tasks in a single rerun vs. CRITIC 4/73, AutoManual 5/73, Reflexion 6/73.
  • Overall accuracy: 55.8% → 63.6% (+7.8 pp), with largest gain on Level-2 multi-hop tasks (48.8% → 61.6%).
  • Token cost: DeepDebug averages 12.8K tokens per trace vs. 8.1K for a single whole-trace pass (1.6Ă— overhead).
  • Ablation: replacing structure-guided reading with a second global search costs 4.8 strict points on gpt-5.4-mini.
  • DeepDebug's margin concentrates on traces >40 events (the regime its structure-guided and cross-examination stages target); on shorter traces, single-pass methods are competitive.

Limitations

  • On hosted LLM backbones (Appendix C), a single global reading is already stronger and DeepDebug's adjudication provides no improvement, motivating per-model routing rather than unconditional escalation.
  • The structure-guided advantage is demonstrated on only 26 traces (>40 events) and is treated as descriptive evidence, not a statistically powered result.
  • Error Hub long-term memory effect (retrieval of similar past cases seeding hypotheses) is explicitly noted as not yet evaluated.
  • Automatic rerun of arbitrary external tools from imported logs is unsupported.
  • Pattern-based redaction in the Error Hub scrubber cannot guarantee removal of all sensitive content; sharing remains opt-in and requires human review.
  • DeepDebug calls (\(\approx 5\) LLM turns) add latency; the escalation policy is heuristic, not learned.

Relevance to Agentic AI / LLM Agents

AgentDebugX directly addresses the reliability and maintainability gap in deployed LLM agent systems, a growing concern as agents operate over long horizons with tool use, memory, and multi-agent coordination. The paper demonstrates that root-cause localization is a prerequisite for effective self-correction—targeted, evidence-backed fixes repair 2–3× more failures than generic reflection—establishing attribution as a first-class component of agentic pipelines rather than an offline analysis. The closed-loop Detect → Attribute → Recover → Rerun architecture and shared portable trace format provide a practical substrate for benchmarking, regression testing, and community-scale incident sharing, directly enabling more systematic study of agent robustness. The installable agentic skill interface also shows that debugging capability can itself be an agent-callable tool, supporting self-healing agent architectures.