Skip to content

AgentAuditor: Human-level Safety and Security Evaluation for LLM Agents

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentAuditor is a training-free, memory-augmented reasoning framework that upgrades LLM-as-a-judge evaluators to human-expert-level accuracy for assessing the safety and security of LLM agents. It combines structured semantic feature extraction, FINCH-based representative shot clustering, auto-generated chain-of-thought reasoning traces, and multi-stage RAG retrieval. Accompanying it is ASSEBench, the first benchmark jointly covering agent safety and security with 2,293 annotated interaction records.

Problem

Existing automated evaluators—both rule-based (keyword/pattern matching) and LLM-based (prompt engineering or fine-tuning)—fail to reliably judge the step-by-step behaviors of agentic systems: they miss cumulative risks across multi-turn interactions, mishandle ambiguous borderline cases, and align poorly with human expert judgment. Prior benchmarks (ToolEmu, AgentSafetyBench, AgentSecurityBench) only test an agent's own risk management, not the evaluator's ability to judge that behavior. No comprehensive benchmark existed to evaluate LLM-based evaluators across both safety and security dimensions simultaneously.

Method

AgentAuditor operates in three stages, all using a single shared LLM:

  1. Feature Memory Construction: For each raw agent interaction record, the LLM extracts structured semantic features—application scenario (s), risk type (r), and behavior mode (b)—via adaptive prompting. Content and features are then embedded with Nomic-Embed-Text-v1.5 into five vector representations (e_c, e_s, e_r, e_b), forming a dual structured+vectorized feature memory FM.

  2. Reasoning Memory Construction: Feature vectors are L2-normalized, weighted, concatenated, and dimensionality-reduced via PCA (variance-preserving threshold). FINCH unsupervised clustering (targeting ~10% of dataset size) identifies the most representative shots (cluster centroids). The LLM then generates high-quality CoT reasoning traces for each representative shot using ground-truth labels and a CoT prompt template, yielding reasoning memory RM.

  3. RAG-based Memory-Augmented Reasoning: For a new query, AgentAuditor first retrieves the top-n candidates from RM by content-embedding cosine similarity, then re-ranks them via weighted feature-embedding similarity (scenario, risk, behavior). The top-k retrieved CoT examples construct a few-shot CoT prompt, which is fed to the LLM to produce a final safe/unsafe judgment.

ASSEBench is built from 13,587 interaction records generated by running four benchmarks (AgentSafetyBench, AgentSecurityBench, AgentDojo, AgentHarm) against three LLMs (Gemini-2.0-Flash-thinking, GPT-4o, Claude-3.5-Sonnet), filtered to 2,293 records via balanced allocation, human expert annotation (binary safe/unsafe + ambiguous flag), and a human-computer collaborative classification scheme covering 15 risk types, 29 scenarios, and 26 behavior modes. Ambiguous records are separately labeled under strict and lenient criteria.

Key Contributions

  • AgentAuditor: a universal, training-free, memory-augmented LLM-as-a-judge framework combining structured feature extraction, FINCH clustering, auto-CoT generation, and multi-stage RAG.
  • ASSEBench: 2,293-record benchmark with four subsets (Security, Safety, Strict, Lenient) jointly covering agent safety and security, with dual-standard annotation for ambiguous cases—the first of its kind.
  • Systematic analysis of evaluation failure modes across rule-based and LLM-based evaluators in agentic contexts.
  • Demonstrated human-level evaluation accuracy (96.1% Acc on R-Judge, 93.2% on ASSEBench-Security) without any fine-tuning.

Results

  • R-Judge: Gemini-2 + AgentAuditor achieves 96.31 F1 / 96.10 Acc, a +17.1% / +18.3% improvement over the base model; surpasses ShieldAgent by 15.1% F1.
  • ASSEBench-Safety: Gemini-2 gains +48.2% F1 (61.79 → 91.59); Llama-3.1-8B accuracy improves from 54.20% to 81.03% on ASSEBench-Security (+49.5%).
  • Human-level comparison: On R-Judge, AgentHarm, and ASSEBench-Security, AgentAuditor (Gemini-2) achieves 96.1%, 99.4%, 93.2% accuracy vs. single human annotator averages of 95.7%, 99.3%, 94.3%.
  • Universal gains: Improvement is consistent across all 10 base models (including open-source 7B–32B models and proprietary reasoning models) on all five benchmarks.
  • Strict/Lenient adaptability: GPT-4.1's performance gap between ASSEBench-Strict and ASSEBench-Lenient narrows from 10.3% to 5.6% with AgentAuditor; Gemini-2's gap narrows from 13.2% to −2.8%.
  • Ablation: Removing any single component (feature tagging, clustering, few-shot, or CoT) degrades performance; CoT alone or few-shot alone each underperforms the base model, demonstrating synergistic dependency.

Limitations

  • Label quality dependency: noisy ground-truth labels in reasoning memory degrade improvement.
  • Inherits CoT-based method costs: requires reasoning-capable LLMs and higher inference compute than direct prompting.
  • Several heuristic parameters (weights w_c, w_s, w_r, w_b; cluster size target; n, k) were set empirically and lack theoretical grounding.
  • ASSEBench and experiments are binary safe/unsafe only; finer-grained label taxonomies are untested.
  • English-only benchmark; multilingual agentic safety risks are not covered.

Relevance to Agentic AI / LLM Agents

Reliable automated evaluation is a foundational bottleneck for safe deployment of LLM agents: without accurate judges, neither safety training nor red-teaming can close the loop. AgentAuditor directly addresses the evaluator reliability problem—a meta-level challenge that underlies all agent safety research—by enabling any capable LLM to serve as a human-caliber judge without task-specific fine-tuning. ASSEBench fills an important infrastructure gap by being the first benchmark that covers both safety (unintentional agent failures) and security (adversarial attacks like prompt injection and data poisoning) with nuanced ambiguity handling, enabling more rigorous comparison of future evaluator methods. The RAG+CoT+clustering architecture also demonstrates a reusable pattern for injecting structured experiential reasoning into LLM judges, relevant to any agentic evaluation pipeline.