Skip to content

Automated Benchmark Auditing for AI Agents and Large Language Models

🕒 Published (v1): 2026-05-25 17:44 UTC · Source: Arxiv · link

Why this paper was selected

Automated benchmark auditing finds latent bugs in expert-authored agent tasks at scale

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Auto Benchmark Audit (ABA) is an agentic framework that systematically audits AI benchmarks by detecting instruction ambiguities, environment conflicts, and evaluation logic failures at task level. Applied to 168 benchmarks and 34,285 tasks, it finds major issues in 25.7% of tasks. Filtering flagged tasks shifts model rankings and boosts average performance on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%, respectively.

Problem

Modern AI benchmarks — especially agentic ones with containerized execution environments, multi-stage evaluation harnesses, and expert-authored tasks — exceed what generalist human annotation can reliably verify. Domain experts leave implicit assumptions about environment setup, task constraints, and grading logic that produce silent failures, incorrect capability assessments, and distorted model rankings. No scalable automated pipeline existed to catch these issues across heterogeneous benchmark formats.

Method

ABA is a two-agent pipeline:

  1. Evidence Collector Agent: Downloads the benchmark repository/dataset and maps heterogeneous task representations (file trees, JSON patches, QA pairs, container images) into a uniform manifest schema with fields instruction_text, solution_ref, environment_ref, tests_ref, and optional trajectory paths. New benchmarks require only a name and URL; no per-benchmark prompt modifications are needed.

  2. Auditor Agent: Initialized with the manifest, it inspects artifacts via shell tools under a standardized rubric across three orthogonal issue axes — Instruction (underspecification, ambiguity), Environment (missing dependencies, container conflicts), and Evaluation (narrow/broad/misaligned grading, incorrect ground truth). Each finding carries: category, severity \(\in \{0, 1, 2\}\), evidence (file paths), claim + downstream failure mode, and a suggested fix.

Two operating modes: static (task definition only) and trajectory (additionally reads recorded agent traces and test outputs), with trajectory mode flagging 8.5% more major tasks on average. The framework runs via Claude Code CLI (Opus 4.7, default tool permissions).

Key Contributions

  • Benchmark-agnostic agentic auditing pipeline covering static-QA, patch-based, and full agentic benchmarks under a single schema.
  • Large-scale audit of 168 benchmarks / 34,285 tasks across 9 domains; all findings, per-task evidence, and rubric annotations released publicly.
  • Empirical validation via (a) independent maintainer fix PRs (Terminal-Bench 2: 66.7% strict recall), (b) comparison against BenchGuard gold sets (BixBench: 62.5% strict recall vs. 54.2% baseline; ScienceAgentBench: 83.3% strict recall), and (c) manual expert review (73% strict precision on major issues in static mode; 92% on SWE-bench Verified trajectory mode).
  • Quantification of downstream distortion: removing flagged tasks increases average performance by 9.9% on SWE-bench Verified and 9.6% on Terminal-Bench 2.

Results

  • Overall issue rate: 25.7% of 34,285 tasks carry major issues (severity 2); 15.1% carry only minor issues; fewer than 60% are clean.
  • Domain extremes: Math (13.2% major, dominated by instruction issues 65%) and Coding (14.1% major) are cleanest; Safety/Alignment (42.2%), Medical (41.5%), and Professional (38.4%) are most problematic.
  • Terminal-Bench 2 (trajectory mode): 66.7% strict / 81.0% partial recall vs. maintainer fix PR ground truth; 58.3% / 70.8% precision (lower bound — PR is non-exhaustive).
  • BixBench vs. BenchGuard baseline: ABA strict recall 62.5% vs. 54.2%; precision 50.0%/71.4% vs. 38.7%/67.7%.
  • ScienceAgentBench vs. BenchGuard baseline: matched at 83.3% recall; ABA reduces redundancy from 1.64× to 1.40×.
  • Manual confirmation (static, 56 major issues): 73% strict / 91% partial precision.
  • Trajectory vs. static: trajectory mode finds +8.5% more major tasks and +6.1% more non-clean tasks on average across 8 shared benchmarks.
  • Leaderboard distortion: removing flagged tasks shifts model rankings and raises mean scores by ~10 percentage points on two frontier coding/agentic benchmarks.

Limitations

  • Precision on major issues in static mode is 73% strict — nearly 1 in 4 flagged major issues may be false positives.
  • Trajectory mode requires pre-recorded agent traces and test outputs, which are unavailable for most benchmarks at audit time.
  • Budget constraints limit task coverage within each benchmark; the audit is not exhaustive within any single benchmark.
  • Validation ground truth is extremely scarce (only three benchmarks have curated gold-issue sets); most domain-level precision is unverified by independent human review.
  • The auditor is itself a large LLM (Opus 4.7/4.6), introducing dependence on model capability and susceptibility to the same specification gaps it is auditing.
  • Framework excludes subjective evaluations, embodied AI, audio, and remote sensing; issue rates in those domains are unknown.

Relevance to Harnesses / Meta-Harnesses

ABA is itself a meta-harness: it operates over evaluation harnesses rather than within them, consuming heterogeneous benchmark artifacts (containers, JSON datasets, test suites) and producing structured audit records at task level. The evidence collector's manifest abstraction is a direct meta-harness pattern — normalizing incompatible harness schemas into a single interface so a uniform auditor agent can operate benchmark-agnostically across 168 distinct evaluation environments. For researchers building or studying harnesses, ABA demonstrates that agentic meta-layers can surface harness defects (brittle grading logic, undeclared environment dependencies) that no static checker or human reviewer catches at scale. The trajectory mode specifically shows the value of plugging a meta-auditor into an execution harness's output stream rather than inspecting it offline.