Skip to content

GuardianAgentBench: Where Agents Fail and How to Guard Them

🕒 Published (v1): 2026-07-23 07:05 UTC · Source: Arxiv · link

Why this paper was selected

580-scenario safety benchmark for autonomous agents; maps failure modes and guards—directly useful for harness evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GuardianAgentBench (GABench) is a 580-scenario benchmark evaluating LLM agent safety on three production frameworks (LangChain, LlamaIndex, Vectara) under five adversarial attack modes. Even the strongest tested configuration (Claude Opus 4.5 on Vectara) achieves only 74.8% overall accuracy. A proof-of-concept execution-time guardrail layer recovers 19.9% of failures at a false positive rate of 0.5%, outperforming system-prompt-based defenses across all models.

Problem

Existing agent safety benchmarks evaluate agents in simulated or custom environments rather than production frameworks practitioners actually deploy; rely on limited automatic validation without sufficient human examination; and do not evaluate guardrails on a common testbed, leaving practitioners without actionable defense guidance. This leaves a gap between research findings and real-world deployability.

Method

Benchmark construction follows a five-stage LLM pipeline per scenario: (1) user intent generation from agent config, (2) self-contained and incomplete prompt generation, (3) realistic tool response simulation (exact/semantic match strategies), (4) ground-truth execution trace construction as a DAG specifying tool call ordering and data flow, and (5) response evaluation criteria definition. Adversarial variants are created by modifying tool responses across five modes: Massive Data (500–1500 word expansions), Error Conditions (realistic service failures), Multiple Matches (3–5 ambiguous alternatives), Prompt Injection (embedded instruction overrides/jailbreaks), and Partial Data (missing fields/truncated content). Scenarios are validated through iterative human annotation (two independent annotators per scenario; three refinement cycles).

Evaluation uses a Claude Sonnet 4.5 judge (93.3% alignment with human annotations on 60 cases) scoring Response correctness and Action correctness independently; Overall requires both. Failures are classified into five types: Incorrect Tool Selection (ITS), Invalid/Missing Parameters (IMP), Missing Required Tool Call (MTC), Repeated Tool Calls (RTC), and Incorrect Tool Order (ITO).

Guardrails are implemented in LlamaIndex as pre-execution interceptors that issue Pass / Provide Corrective Feedback / Block verdicts. Three guardrails run concurrently via asyncio.gather: Argument Validation, Tool Coverage Check, and Relevance and Cost Check. Corrective feedback triggers up to two retry iterations; exhausted retries escalate to a human-in-the-loop alert.

Key Contributions

  • GABench: first agent safety benchmark evaluated on production-ready agentic frameworks (LangChain, LlamaIndex, Vectara), covering 580 scenarios, 81 unique tools, 1,177 sequential turns, six domains, and five adversarial attack modes.
  • Empirical characterization of two qualitatively distinct failure regimes: stronger models under-call required tools (MTC: 52–57% of failures); weaker models mis-select and over-call (ITS+RTC: up to 58% combined).
  • Quantified complexity bottlenecks: performance degrades monotonically with tool-set size (−15.9 points, 1→7 tools) and sequential turn depth (−31.1 points, 1→7 turns), with long-horizon planning being the steeper bottleneck.
  • Proof-of-concept execution-time guardrail layer outperforming system-prompt-based defenses by +2.8 to +7.7 points across all six models at 19.9% failure recovery and 0.5% FPR.

Results

  • Best overall configuration: Claude Opus 4.5 on Vectara, 74.8% Overall — agents fail roughly one in four scenarios.
  • Model tier ranking (avg Overall): Claude Opus 4.5 / Gemini-3-Pro (71–75) > GPT-5.2 Pro / Qwen3-Max (68–72) > DeepSeek-V3.2 / GPT-OSS-120B (63–68).
  • Hardest domain: Calendar (no model exceeds 62.0 Overall); easiest: Financial and Customer Service.
  • Framework impact: within each model, scores vary ≤2–3 points across LlamaIndex / LangChain / Vectara — gaps are model-driven, not framework-driven.
  • Tool-set scaling (Claude Opus 4.5): Overall drops from 78.2 → 62.3 (1→7 tools, −15.9 pts).
  • Turn-depth scaling (Claude Opus 4.5): Overall drops from 82.3 → 51.2 (1→7 turns, −31.1 pts).
  • ITO (Incorrect Tool Order) is consistently the rarest failure type across all models (0.6–4.4%), suggesting tool ordering is a largely solved problem.
  • MTC (Missing Required Tool Call) dominates for stronger models (GPT-5.2 Pro/GPT-OSS-120B: 55–57%; Gemini-3-Pro: 52–55%); weaker models show high RTC (29–33%) and ITS (14–25%).
  • Guardrails vs. system-prompt baseline: +2.8 to +7.7 points improvement across all six models; 19.9% failure recovery at 0.5% false positive rate.

Limitations

  • Guardrail implementation is proof-of-concept only, integrated solely into LlamaIndex; cross-framework guardrail generalization is not demonstrated.
  • Scenario generation and judging both rely on Claude Sonnet 4.5, introducing potential model-specific biases and circular evaluation artifacts.
  • 580 scenarios may be insufficient to fully characterize the long tail of failure modes across all adversarial conditions.
  • Adversarial attack modes are applied to tool responses only, not to user prompts or system prompts directly (except Prompt Injection, which embeds instructions in tool response fields).
  • The benchmark covers six domains but cannot claim comprehensive coverage of real-world agentic deployments; domain selection may not generalize.
  • Human validation was conducted by in-house annotators; independent third-party annotation is not reported.

Relevance to Agentic AI / LLM Agents

GABench directly advances the rigorous evaluation of tool-using LLM agents in realistic deployment conditions, moving safety benchmarking from simulated sandboxes to the production frameworks (LangChain, LlamaIndex, Vectara) actually used in practice. The finding that capability tier determines the character of failure — not just its frequency — has concrete implications for agent architecture: stronger models need better tool-coverage enforcement, while weaker models need stronger selection and deduplication mechanisms. The execution-time guardrail paradigm (intercept → structured feedback → retry → escalate) demonstrates a composable safety layer orthogonal to model alignment that can be layered over any agent without retraining, which is directly actionable for practitioners building production agentic systems. The quantified bottleneck from long-horizon planning (−31.1 pts over 7 turns) provides a concrete difficulty axis for future agent benchmarks in this line of work.