Skip to content

UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

🕒 Published (v1): 2026-07-09 17:59 UTC · Source: Arxiv · link

Why this paper was selected

Benchmark for proactive real-world agents; evaluates harness behavior across tool-use scenarios

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UniClawBench is a 400-task capability-driven benchmark for proactive agents running in live Docker environments, structured around five foundational capability axes rather than application domains. It introduces a three-role closed-loop evaluation framework—executor, hidden supervisor, and user simulator—that enables multi-turn feedback without leaking grading criteria. The key finding is that agent framework choice has a larger impact on performance than base model choice.

Problem

Existing agent benchmarks rely on sandboxed/cached environments (e.g., WebArena, OSWorld), use single-turn evaluation paradigms that ignore iterative human feedback, and organize tasks by application scenario (e.g., "office", "travel"), which conflates multiple model capabilities and makes it impossible to pinpoint the root cause of failures. Additionally, live environments lack stable ground truth because real-world content (prices, pages) changes over time.

Method

UniClawBench decomposes agent evaluation into five capability dimensions—Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination—with 80 tasks per dimension (40 English, 40 Chinese), all executed inside Docker containers with live browsers, file systems, and installed tools.

Three-role closed-loop evaluation: 1. Executor Agent: runs the task using only public inputs (tools, files, web) and produces trajectories + artifacts. 2. Supervisor Agent (hidden): applies fine-grained, per-checkpoint rubrics against private references; outputs a structured state \(\{pass, fail, continue\}\) and a weighted score. Checkpoints are weighted, e.g., \(C_5\) (missing-ingredient accuracy) carries 0.30 while \(C_1\) (official video source) carries 0.10. 3. User Simulator (LLM-based): receives only the visible trajectory and a coarse progress signal (not the rubric) through an information firewall; generates natural-language feedback to the executor, enabling multi-turn interaction without answer leakage.

Termination: task passes if final score \(\geq\) threshold (e.g., 0.9); continues if recoverable and follow-up turns remain (up to 3); fails otherwise or on timeout (30 minutes). Metrics are Pass Rate (PR) (task-level binary) and Average Score (AS) (step-level aggregated checkpoint score).

Cross-framework evaluation covers OpenClaw, EDICT, and Nanobot under models GPT-5.4, Claude Opus-4.8, and Kimi-2.6.

Key Contributions

  • First capability-driven benchmark for proactive agents in dynamic real-world settings, organized by foundational capability bottleneck rather than application scenario.
  • Three-role closed-loop evaluation with an information firewall that supports authentic multi-turn interaction without leaking grading criteria or ground truth.
  • Hidden checkpoint rubrics that replace static pre-recorded answers, enabling evaluation in live, mutable environments.
  • Joint cross-model Ă— cross-framework evaluation that disentangles base model capability from framework-level design choices.
  • 400 bilingual (English + Chinese) manually constructed tasks spanning diverse input/output modalities (live web, video, audio, PDFs, GUI application state, calendar files, databases).

Results

  • Framework choice consistently impacts capability performance more than model choice across all five capability dimensions.
  • Best observed normalized pass rate: Claude Opus-4.8 + OpenClaw achieves ~0.69.
  • Long-context and multimodal tasks are the primary bottlenecks for all model–framework combinations.
  • Cross-framework comparisons across OpenClaw, EDICT, and Nanobot show architecture-level design has a stronger effect than swapping GPT-5.4 / Claude Opus-4.8 / Kimi-2.6.

(Note: fine-grained per-capability numeric breakdowns by model and framework are not reported in the provided text excerpt.)

Limitations

  • Only three agent frameworks (OpenClaw, EDICT, Nanobot) and three frontier models evaluated; generalization to other frameworks/models is unverified.
  • The user simulator is LLM-based and may not perfectly replicate diverse real human feedback styles.
  • 400 tasks is relatively small, and live web content means tasks may degrade over time as external resources change.
  • The information firewall relies on the user simulator not reasoning backward from coarse signals; this isolation may not be perfect in future more capable models.
  • Benchmark focuses on a single-primary-capability design; tasks requiring genuinely equal contributions from multiple capabilities are excluded by construction.

Relevance to Harnesses / Meta-Harnesses

UniClawBench's three-role closed-loop evaluation strategy is itself a meta-harness: it orchestrates an executor agent, a hidden supervisor agent, and a user simulator agent within a Docker-based runtime, coordinating their information flow through an explicit firewall. The finding that framework choice dominates model choice is directly actionable for harness designers—it implies that how a meta-harness wraps and scaffolds a model (tool routing, memory, multi-turn state management) matters more than which underlying LLM is selected. The hidden-supervisor pattern, where grading criteria are isolated from the agent under test, is a reusable architectural primitive for any harness that must evaluate agents in live, non-deterministic environments.