Skip to content

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

🕒 Published (v1): 2026-05-11 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Real-world long-horizon CLI harness benchmark against synthetic sandboxes and mock servers

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WildClawBench is a 60-task, native-runtime benchmark for evaluating long-horizon CLI agents inside reproducible Docker containers running actual harnesses (OpenClaw, Claude Code, Codex, Hermes Agent) with real tools. Across 19 frontier models the best score is 62.2% (Claude Opus 4.7), and switching harness alone shifts a single model by up to 18 points. The benchmark exposes that the scaffold is as much a performance determinant as the underlying model.

Problem

Existing agent benchmarks rely on synthetic sandboxes, short-horizon tasks (< 1 minute), mock-service APIs, and final-answer-only checks. They cannot reveal whether agents succeed through reliable, auditable interaction with actual runtimes, nor whether results transfer across the CLI harnesses used in real deployment.

Method

Each task is a Markdown specification pairing YAML metadata, an agent-facing prompt, expected behaviors, human-readable rubrics, and an executable grading function. Tasks run in isolated Docker containers pre-loaded with a specific harness; ground-truth assets are injected only after the agent exits to prevent leakage. All 19 models are routed through a unified OpenRouter endpoint, with tool schemas and system prompts held constant within each harness. Grading is hybrid: (1) deterministic rule-based checks (file existence, format, numerical accuracy, string matching); (2) environment-state auditing of side effects via instrumented service audit logs and transcript inspection; (3) LLM/VLM-as-judge (GPT 5.4) for outputs that defy exact matching (narrative reports, images, video, malicious-intent detection). The four-stage curation pipeline (Authoring → Reference Construction → Filtering → Refinement) enforces a discriminability threshold: a task is retained only if \(\max_{i \neq j} \Delta_{ij} \geq 0.2\) across pilot model scores.

Key Contributions

  • WildClawBench: 60 human-authored, bilingual (36 EN / 24 ZH), multimodal (26 of 60) tasks across six categories averaging ~8 min and >20 tool calls per run.
  • Native-runtime evaluation: agents run inside the actual CLI harness (OpenClaw, Claude Code, Codex, Hermes Agent) rather than mock APIs, enabling harness-level ablations.
  • Hybrid grading: first benchmark to combine rule checks, environment-state auditing, and LLM/VLM judging with containerized reproducibility and bilingual coverage simultaneously.
  • Harness-as-variable: systematic comparison of four harnesses on identical tasks and models, quantifying scaffold contribution to agent capability.
  • Safety-in-workflow tasks: 10 tasks embed prompt injection, malicious skill injection, dangerous OS commands, and credential-leak detection inside otherwise normal workflows.

Results

  • Best overall score: Claude Opus 4.7 at 62.2% under OpenClaw; every other model stays below 60%; scores span 19.3%–62.2% (43-point range).
  • Pure-text vs. multimodal gap is common: GPT 5.4 scores 58.0% (text) vs. 40.2% (multimodal); Claude Opus 4.7 scores 65.0% vs. 58.5%.
  • Harness effect: MiMo V2 Pro shifts 18 points between Claude Code (29.9%) and Hermes Agent (48.1%); GLM 5 and MiMo V2 Pro each lose >10 points on Claude Code vs. OpenClaw.
  • Codex harness yields the highest scores for 3 of 4 tested models (e.g., GPT 5.4: 56.8% on Codex vs. 50.3% on OpenClaw); Hermes Agent is best for three models overall.
  • Claude Code is the most latency-bound harness (9.1–10.2 min/task), leading to more timeout-terminated trajectories.
  • Cost efficiency varies independently of accuracy: DeepSeek V4 Pro reaches 43.7% at $0.20/task vs. Claude Opus 4.7 at 62.2% for $1.29/task.

Limitations

  • Only 60 tasks; small task count limits statistical power, especially within individual categories (as few as 6 tasks for Social Interaction).
  • Human expert curation (8 researchers, 2 weeks) makes the benchmark expensive to extend or update.
  • The discriminability filter (\(\Delta_{ij} \geq 0.2\)) biases against tasks that all models uniformly fail or pass, potentially pruning extreme-difficulty items.
  • LLM/VLM judging uses GPT 5.4 as the judge, introducing model-specific bias and potential version-drift over time.
  • Harness evaluation is limited to four harnesses; findings may not generalize to emerging or proprietary scaffolds.
  • Performance variance across repeated runs is noted but relegated to a supplementary table (Table 10), leaving reliability of point estimates under-characterized in the main text.

Relevance to Harnesses / Meta-Harnesses

WildClawBench is one of the first benchmarks to treat the harness itself as an independent experimental variable, demonstrating empirically that control-loop design, tool schemas, context-management policy, and output-recovery logic jointly determine effective agent capability — sometimes more than model choice. The up-to-18-point swing from harness substitution provides quantitative grounding for the thesis that harness architecture is a first-class engineering concern, not a neutral wrapper. For researchers designing or evaluating meta-harnesses (systems that orchestrate or select among lower-level harnesses), the benchmark offers both a methodology (containerized, harness-swappable evaluation) and a set of task categories (including productivity-flow tasks that explicitly model paper-digest and multi-source aggregation workflows) directly applicable to testing harness-level abstractions. The hybrid grading framework — especially environment-state auditing of side effects — also exemplifies how meta-harness correctness should be measured beyond final-answer checks.