UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks¶

🕒 Published (v1): 2026-07-09 17:59 UTC · Source: Arxiv · link

Why this paper was selected

UniClawBench: universal benchmark for proactive agents on real-world tasks — fills evaluation gap for agentic systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

UniClawBench is a capability-driven benchmark of 400 bilingual real-world tasks for evaluating proactive LLM agents across five foundational ability dimensions, executed in live Docker environments rather than sandboxed replicas. It introduces a three-role closed-loop evaluation strategy (executor, hidden supervisor, user simulator) with an information firewall to enable realistic multi-turn feedback without leaking grading criteria. Empirically, framework choice is found to affect performance more than model choice, with long-context and multimodal tasks as the primary bottlenecks.

Problem¶

Existing agent benchmarks exhibit three structural limitations: (1) they use sandboxed or cached environments (e.g., WebArena, OSWorld) that diverge from real-world conditions; (2) they employ single-turn evaluation, ignoring the iterative, closed-loop nature of human-agent interaction; (3) they organize tasks by application domain (e.g., "office," "travel"), which conflates distinct model capabilities and prevents pinpointing root causes of failures (e.g., whether a breakdown stems from visual perception, long-context reasoning, or tool misuse).

Method¶

UniClawBench decomposes evaluation across five capability dimensions, each designed so that the primary bottleneck is a single well-defined ability:

Skill Usage — invoke and correctly apply a declared tool/API (OCR, SQL, Git, audio transcription) to produce a structured artifact.
Exploration — open-ended investigation under incomplete or misleading information (reverse-engineer undocumented APIs, trace image sources, resolve configuration conflicts).
Long-Context Reasoning — aggregate and reconcile evidence across heterogeneous sources (multiple PDFs, email threads, video + web pages) while maintaining global consistency.
Multimodal Understanding — extract and act on evidence exclusively available in images, video, or audio.
Cross-Platform Coordination — preserve and transfer state across heterogeneous applications (browser, GUI apps, calendar, citation manager, local filesystem).

Tasks run inside Docker containers with real browsers, filesystems, and installed software; dynamic content (e.g., live product prices) is handled via step-by-step completion checkpoints rather than pre-recorded answers. A three-role closed-loop evaluation is used: (1) the Executor Agent performs the task using an agent framework (OpenClaw, Nanobot, or EDICT); (2) the Supervisor Agent (hidden) applies weighted checkpoint rubrics against private references, producing a structured state (pass/fail/continue) and a score; (3) the User Simulator receives only the visible trajectory and a coarse progress signal (an information firewall blocks rubric details), then generates natural-language corrective feedback for the executor. Up to three multi-turn follow-up rounds are allowed before final termination. Metrics are Pass Rate (PR, task-level binary) and Average Score (AS, weighted checkpoint aggregate).

Key Contributions¶

First capability-oriented benchmark for proactive agents evaluated in live Docker environments, with 400 bilingual (EN/ZH) tasks spanning 11 application domains.
Three-role closed-loop evaluation with strict information firewall enabling multi-turn user feedback without criterion leakage or solution contamination.
Cross-framework evaluation (OpenClaw, EDICT, Nanobot) crossed with cross-model evaluation (GPT-5.4, Claude Opus-4.8, Kimi-2.6) to disentangle base model capability from agent architecture design.
Empirical finding that agent framework choice has a stronger effect on capability-level performance than base model choice.

Results¶

Best overall: Claude Opus-4.8 + OpenClaw, normalized pass rate ≈ 0.69.
Framework choice consistently dominates over model choice across capability dimensions.
Long-context and multimodal tasks are identified as the two primary capability bottlenecks across all model-framework combinations.
Three frameworks evaluated: OpenClaw, EDICT, Nanobot; three models: GPT-5.4, Claude Opus-4.8, Kimi-2.6 (paper text truncated before full per-capability breakdown tables).

Limitations¶

Paper text is truncated; per-capability and per-framework numerical breakdowns are unavailable from the provided excerpt.
400 tasks (80 per dimension, 40 per language) is relatively modest scale; domain and scenario diversity may still introduce residual bias.
Live web content introduces inherent non-determinism; checkpoint rubrics mitigate but cannot fully eliminate evaluation variance across runs.
Docker isolation, while closer to real-world than VMs with cached pages, still does not replicate full production environments (e.g., OS-level authentication, persistent account state).
Evaluation relies on LLM-based supervisor and user simulator agents, introducing model-in-the-loop noise and potential blind spots.

Relevance to Agentic AI / LLM Agents¶

UniClawBench directly targets the evaluation gap for modern proactive agents that must handle multi-turn interaction, multimodal inputs, and cross-application orchestration—capabilities central to the current trajectory of LLM agent research. The capability-oriented taxonomy enables failure attribution at a fine granularity that scenario-based benchmarks cannot provide, which is essential for guiding targeted agent improvements. The finding that framework architecture dominates model choice is a high-signal result for the community, suggesting that agent scaffolding (memory, skill management, orchestration loop) is currently a stronger lever than base model scaling for real-world task performance. The information-firewall design for closed-loop evaluation is a methodological contribution applicable to future benchmarks seeking to simulate realistic human-agent collaboration.