Skip to content

AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models

🕒 Published (v1): 2025-09-30 11:20 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgenticEval is a multi-agent framework that autonomously ingests unstructured regulatory documents and generates a continuously self-evolving LLM safety benchmark. Instead of a static test suite, it runs an iterative loop where an Analyst agent learns from evaluation failures and directs a Generator agent to craft progressively harder, targeted test cases. GPT-5's compliance rate on the EU AI Act degrades from 72.50% to 36.36% across three iterations, demonstrating that self-evolving evaluation surfaces vulnerabilities static benchmarks cannot.

Problem

Static safety benchmarks (e.g., HELM, DecodingTrust, StrongREJECT) become obsolete as attack vectors evolve, fail to capture regulatory complexity (EU AI Act, NIST RMF), and cannot be customized to organizational policies. A model deemed safe under a fixed benchmark may remain exploitable against emerging threats or fall out of regulatory compliance.

Method

AgenticEval is a four-agent pipeline built on MetaGPT:

  1. Specialist agent (\(A_S\)) — Parses an arbitrary regulatory document \(R\) into a hierarchical knowledge tree \(\mathcal{K}\) of atomic rules \(r\). Each rule is enriched via web-search-augmented grounding into a Testable Knowledge Base \(\mathcal{K}^*\) containing compliant guidance \(\mathcal{G}_\text{should}\) and adversarial guidance \(\mathcal{G}_\text{should\_not}\).

  2. Generator agent (\(A_G\)) — For each atomic rule \(r \in \mathcal{K}^*\), generates a Question Group \(Q_r\) via a semantic anchor (base open-ended question) plus facet expansion: adversarial jailbreak variants, MCQ/T-F deterministic probes, and multimodal questions with paired images. Initial test suite: \(\{Q_r\}^{(0)}\).

  3. Evaluator agent (\(A_E\)) — Judges each test case \((q, c)\) against target model \(M_\text{target}\) using a principled rubric conditioned on \(c\) and the rule's guidance, yielding binary outcome \(y_q \in \{0,1\}\) plus rationale \(z_q\).

  4. Analyst agent (\(A_A\)) — Takes \(R_r^+\) (passes) and \(R_r^-\) (failures) from the prior round, synthesizes a failure-mode analysis \(z_\text{analysis}\), and emits a new attack strategy \(S_\text{attack}\), which feeds back to \(A_G\) for refined test generation.

This Analyst→Generator→Evaluator loop repeats for \(K_\text{max}\) iterations (set to 3 in experiments), constituting the Self-evolving Evaluation loop.

Key Contributions

  • A new paradigm framing safety evaluation as a continuous, agentic, self-evolving process rather than a one-time static audit.
  • Regulation-to-Knowledge Transformation: automated hierarchical decomposition of arbitrary unstructured policy documents into atomic, testable rules with compliant/adversarial guidance.
  • Question Group construction with systematic facet expansion (jailbreak, MCQ/T-F, multimodal).
  • Self-evolving Evaluation loop driven by an Analyst agent that learns failure modes and escalates attack strategy over iterations.
  • Large-scale evaluation of 11 frontier LLMs (GPT-5, Gemini-2.5-pro/flash, Grok-4, Qwen-3, Llama-4, DeepSeek-V3.1) across three regulatory frameworks (NIST AI RMF, EU AI Act, MAS FEAT).

Results

  • GPT-5 achieves the highest baseline safety across all three frameworks: 78.98% (NIST RMF), 67.16% (EU AI Act), 67.92% (MAS FEAT).
  • GPT-5's EU AI Act compliance drops from 72.50% → 36.36% over three self-evolving iterations, demonstrating progressive hardening.
  • No model shows uniform excellence: GPT-5-chat-latest scores 91.67% on EU AI Act Predictive Policing (PP-RA) but only 44.64% on Real-time Remote Biometric Identification (RRBI).
  • Llama-4-maverick scores 75.00% on NIST Data Privacy Violations but only 26.32% on Intellectual Property Infringement.
  • Smallest models (Qwen-3-8B) score ~39–42% across frameworks; Grok-4 underperforms its tier peers on most categories.
  • Specialist agent validation: cosine-similarity heatmaps of atomic rule embeddings show strong intra-cluster coherence within regulatory dimensions across all three documents.

Limitations

  • \(K_\text{max} = 3\) iterations chosen for cost balance; deeper iteration counts unexplored.
  • Evaluator agent (GPT-4.1) acts as LLM judge — its own biases and failure modes may affect safety rate estimates.
  • Framework costs (multiple frontier models as agents) scale poorly for continuous deployment without budget controls.
  • Self-evolution quality depends on the Generator (Gemini 2.5 Pro) creative capability; diversity of generated attacks is unconstrained by formal coverage guarantees.
  • Only three regulatory frameworks evaluated; generalization to other legal or domain-specific policy texts is assumed but not exhaustively demonstrated.
  • No human-validation pass on generated test cases; ground truth is the policy document's text, not expert legal annotation.

Relevance to Harnesses / Meta-Harnesses

AgenticEval is a meta-harness for evaluation: it wraps multiple specialized agents (Specialist, Generator, Evaluator, Analyst) in an orchestrated pipeline that itself generates, runs, and evolves the benchmarks it uses — a harness whose primary artifact is another harness. The Self-evolving Evaluation loop is structurally a feedback-driven meta-harness: the Analyst agent acts as a harness controller that rewrites test-generation directives based on prior outputs, analogous to how a meta-harness adjusts task decomposition based on subtask results. The implementation on MetaGPT illustrates how a coordinated multi-agent framework can serve as the substrate for higher-order evaluation harnesses. For researchers tracking harness design, this paper concretely demonstrates the pattern of harness self-modification — where the harness observes its own outputs and reconfigures its inner loop rather than being statically defined.