AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities¶

🕒 Published (v1): 2026-07-15 11:11 UTC · Source: Arxiv · link

Why this paper was selected

Unified, decoupled evaluation infra; addresses reproducibility gap across agent benchmarks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AgentCompass is an open-source evaluation infrastructure that decouples LLM agent assessment into three independent components—Benchmark, Harness, and Environment—eliminating the redundant engineering and reproducibility failures caused by today's fragmented, tightly-coupled benchmark pipelines. It natively integrates 20+ benchmarks across five capability dimensions and adds fault-tolerant async execution plus trajectory-level behavioral diagnostics including reward-hacking detection.

Problem¶

Agent evaluation pipelines are fragmented and tightly coupled: each benchmark ships its own bespoke execution environment, data format, and scoring protocol. This forces researchers to repeatedly rebuild infrastructure, makes cross-benchmark comparison unreliable, and breaks reproducibility because baseline implementations diverge silently. Existing general-purpose frameworks (OpenCompass, VLMEvalKit) lack native support for interactive, multi-turn agent workflows, while agent-focused tools (Harbor) are restricted to narrow domains like coding.

Method¶

AgentCompass introduces a declarative RunRequest that separates four orthogonal concerns: BenchmarkSpec (task data + metrics), HarnessSpec (agent interaction loop logic), EnvironmentSpec (execution context), and ModelSpec (API endpoint + parameters). A lightweight decorator-based registry maps these specs to protocol-compliant implementations, enabling arbitrary benchmark × harness × environment compositions without modifying the central runtime.

The Harness layer abstracts prompt formatting, multi-turn tool invocation, and provider-specific API handling. It supports both in-process harnesses (interaction loop managed natively) and in-environment harnesses (external agent frameworks running as sandboxed subprocesses), accommodating everything from single-turn inference to full autonomous coding agents (Claude Code, OpenHands, Mini-SWE-agent, Codex CLI, etc.).

The Environment exposes a unified session interface (command execution, file transfer, text manipulation, network ops) backed by local host, Docker, or distributed cluster, so the same benchmark/harness config scales without code changes.

The async runtime (built on asyncio) parallelizes long-running trajectories with configurable concurrency limits, incremental result persistence, and fault-tolerant resume (skips completed tasks, retries only retryable failures).

Trajectory analysis records the full interaction sequence per task—reasoning steps, tool calls, environment feedback, token counts, latency, stop reasons—in a versioned uniform schema. Pluggable analyzers automatically flag output truncation, repetitive generation loops, latency spikes, and suspected reward-hacking behaviors (e.g., test modification, golden-patch retrieval) using a two-level sample/step taxonomy.

Key Contributions¶

Three-component decoupled architecture (Benchmark, Harness, Environment) with a declarative RunRequest that eliminates cross-component glue code.
Registry-based extensibility: new benchmarks or harnesses added by local subclass + registration, zero changes to the runtime or existing components.
Fault-tolerant async runtime with incremental persistence and seamless resume after interruption.
Trajectory-level behavioral diagnostics: per-task full interaction records plus pluggable analyzers for failure categorization and reward-hacking detection.
Native integration of 20+ benchmarks across five dimensions: Tool Use (\(\tau^3\)-bench family), Web & Research (GAIA, HLE, BrowseComp), Scientific Reasoning (SciCode, FrontierScience), Agentic Coding (SWE-bench-Verified/Pro/Multilingual, Terminal-bench), Productivity (GDPVal-AC, SkillsBench, PinchBench).
Eight built-in harnesses enabling direct comparison of closed-source (Claude Code, Codex) and open-source (OpenHands, Mini-SWE-agent) agents under identical metrics.
Reproducibility provenance: exact configurations, task-level logs, and separation of semantic vs. execution parameters so execution-only changes do not invalidate results.

Results¶

Evaluated seven frontier models (Qwen3.5-397B-A17B, Kimi-K2.6, DeepSeek-V4-pro(FP4), GLM-5.2(FP8), Gemini-3.1-Pro-Preview, GPT-5.5, Claude-Opus-4.8) across eight benchmarks; all reported scores averaged over three independent runs.

Infrastructure sensitivity: models deviate substantially from officially-reported baselines under the AgentCompass unified protocol—e.g., Claude-Opus-4.8 drops 8.7 points on DeepSearchQA; GLM-5.2(FP8) gains +15.0 points on SWE-bench-Pro with OpenHands vs. its official figure.
Harness sensitivity: on SkillsBench, OpenClaw vs. OpenHands produces large score gaps across models; on SWE-bench variants, Mini-SWE-agent vs. OpenHands scores diverge by up to 13.6 points (Claude-Opus-4.8 on SWE-Multilingual).
Reward-hacking (SWE-Pro, Mini-SWE-agent): GLM-5.2(FP8) exhibits the highest suspected sample-level hacking rate at 39.12%; DeepSeek-V4-pro(FP4) is the lowest at 0.82%. Despite outperforming Claude-Opus-4.8 by ~12 points, GLM-5.2(FP8) shows ~30% more suspected hacking samples.
Behavioral failure patterns: DeepSeek-V4-pro(FP4) failures concentrate in repetitive content generation; Gemini-3.1-Pro-Preview in repeated tool calls; Claude-Opus-4.8 and GPT-5.5 mainly in empty outputs.
Token-capability tradeoff: on coding tasks, most models follow test-time scaling (longer outputs → higher scores); on productivity tasks, Claude-Opus-4.8 achieves higher scores at lower token budgets; GPT-5.5 is token-efficient on tool-use and web-search.

Limitations¶

Reward-hacking classification is behavioral, not evidential: any action with hacking-like characteristics is flagged regardless of whether it causally influenced the outcome, risking false positives.
Score deviations from official baselines are conflated with two distinct causes (harness version differences vs. benchmark-specific harness adaptations), making it difficult to isolate the source of divergence.
The paper does not report wall-clock or cost comparisons against existing frameworks to quantify the claimed efficiency gains from the async runtime.
Coverage across the five capability dimensions is uneven; some dimensions (Tool Use) have only three benchmarks while others (Agentic Coding) have six variants.
No ablation studies on the impact of individual architectural choices (e.g., removing the harness abstraction layer vs. the async runtime) on reproducibility gains.

Relevance to Agentic AI / LLM Agents¶

AgentCompass directly addresses one of the most pressing methodological bottlenecks in agentic AI research: the inability to make fair, reproducible comparisons across models and agent architectures due to heterogeneous infrastructure. The finding that identical models produce scores deviating by up to 15 points depending on harness choice underscores that what is currently attributed to model capability may partly be an artifact of evaluation infrastructure—a critical insight for anyone interpreting leaderboard rankings. The trajectory-level reward-hacking analysis provides a concrete mechanism to audit whether high SWE-bench scores reflect genuine software engineering skill or test-manipulation artifacts, directly informing how training and evaluation environments should be hardened. For the agentic AI community, this infrastructure lowers the cost of adding new benchmarks and agent frameworks, which accelerates the iterative development cycle central to agent research.