Skip to content

ClawEnvKit: Automatic Environment Generation for Claw-Like Agents

🕒 Published (v1): 2026-04-20 17:36 UTC · Source: Arxiv · link

Why this paper was selected

Automatic environment generation pipeline for training and evaluating claw-like agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ClawEnvKit is an automated pipeline that generates verified evaluation and training environments for "claw-like" CLI agent harnesses (e.g., OpenClaw, NanoClaw) from natural language descriptions, replacing manual, hours-per-task construction. It produces Auto-ClawEval, the first large-scale benchmark for this agent family (1,040 environments across 24 categories), at 13,800× lower cost than human curation. Across 8 harness frameworks and 4 model families, harness engineering boosts task completion by up to 15.7 percentage points over a bare ReAct baseline.

Problem

Constructing training and evaluation environments for claw-like agents (CLI agents using tool calls and skill markdown) is manual, requires ~2 hours per task, and produces static benchmarks that do not scale with rapidly advancing agent capabilities. Existing benchmarks (Claw-Eval: 104 tasks, ClawArena: 64 tasks) are human-curated and cannot adapt to new tasks, long-tail domains, or an agent's current weaknesses.

Method

ClawEnvKit formalizes each environment as a triple \(E = (P, M, C)\): task specification \(P \in \mathcal{L}\), interaction interface \(M = (\mathcal{T}, O)\) (callable tools + audit log), and evaluation functional \(C = \{(c_i, w_i)\}\) scoring trajectories via \(R(\sigma, E) = \sum_i w_i \cdot c_i(\sigma, O)\).

Three LLM-based modules instantiate this triple from a natural language prompt \(\phi\): 1. Parser — single LLM call decomposing \(\phi\) into typed intent units (actions, objects, constraints) that map directly to elements of \((P, M, C)\). 2. Generator — three sub-workflows: (a) task generation from intent units with diversity controls (action rotation, deduplication); (b) service generation, which designs and tests new mock API servers on demand; (c) fixture generation, which mounts task-specific data (databases, images, documents) into isolated containers. 3. Validator — three sequential checks: Format Check (field completeness, scoring weights sum to 1, no self-contradictions), Coverage Check (every intent unit appears in \(P\), \(M\), or \(C\)), Feasibility Check (LLM call for counterfactual detection). Failed environments retry up to 3 times before being discarded.

Execution runs each task in an isolated sandbox with no internet access; mock services inject random API errors on 25% of calls to test robustness. Harness integration operates across three tiers: native tool plugins (OpenClaw), MCP servers (Claude Code, Codex, NanoClaw, etc.), and curl-based SKILL.md prompt injection (CoPaw, NemoClaw, Hermes). Grading uses server-side audit logs — not agent self-reports — with 15 check types; LLM-judge weight is capped at 55% to ensure majority-deterministic scoring. The composite score follows \(R(\sigma, E) = \text{safety}(\sigma) \times (0.8 \cdot \text{completion}(\sigma, C) + 0.2 \cdot \text{robustness}(\sigma, M))\).

Key Contributions

  • ClawEnvKit pipeline: end-to-end automated generation of verified agent environments from natural language, with isolated sandboxes and harness-agnostic execution.
  • Auto-ClawEval: first large-scale cross-harness benchmark for claw-like agents — 1,040 environments across 24 semantic categories, generated for ~$80 in API costs using claude-sonnet-4.6.
  • Auto-ClawEval-Mini: 104-task compact benchmark paired 1:1 with human-curated Claw-Eval for controlled quality comparison.
  • Live evaluation: on-demand environment generation from user-provided natural language, enabling continuous and long-tail evaluation without human labor.
  • On-demand training generator: produces task distributions that adapt to an agent's current weaknesses rather than being bounded by existing user logs.

Results

  • Auto-ClawEval-Mini matches or exceeds Claw-Eval on all quality dimensions: Coherence 0.59 vs 0.51, Clarity 3.54 vs 3.38, Validity 100% for both; construction time reduced from 208 h (human) to 1.8 h (automated).
  • 1,040-environment Auto-ClawEval generated at ~$80 API cost (13,800× cost reduction implied).
  • Scores on Auto-ClawEval vs Auto-ClawEval-Mini differ by <2% across all models and harnesses, validating the mini benchmark as a reliable proxy.
  • Harness engineering boosts performance by up to 15.7 pp: ReAct Agent Loop baseline 53.3% vs. best harness (NemoClaw) 69.0%.
  • Safety and robustness scores are near-ceiling (≥83%) across all configurations; completion ranges 34%–76%, confirming the benchmark is unsaturated by frontier models.

Limitations

  • Auto-ClawEval covers only 15 mock services (vs. Claw-Eval's 19), potentially narrowing task diversity.
  • Generator retries up to 3 times then discards; failure rate and its distributional impact on task categories are not reported.
  • LLM-judge component (capped at 55% weight) introduces model-dependent grading variance; no inter-rater reliability analysis is presented.
  • Benchmark focuses on claw-like/CLI agents with structured tool calls; generalization to GUI agents or open-ended web agents is not demonstrated.
  • Task difficulty and category balance are controlled by natural language prompts to the Generator, with no formal guarantees on coverage uniformity.

Relevance to Harnesses / Meta-Harnesses

ClawEnvKit is directly relevant as a meta-harness evaluation framework: it systematically benchmarks 8 agent harness frameworks side-by-side (OpenClaw, NanoClaw, IronClaw, Claude Code, Codex, Cursor, CoPaw, NemoClaw, and others) within a unified, reproducible scoring environment, producing the first empirical evidence that harness engineering — not just model quality — drives a 15.7 pp performance gap. The three-tier harness integration strategy (native plugin / MCP server / SKILL.md injection) is itself a meta-harness abstraction that adapts the evaluation scaffold to each agent's native workflow without modifying the underlying tasks or graders. For researchers tracking harness design, this provides a scalable testbed to measure how harness architecture choices translate to downstream task performance, and the live-evaluation capability enables continuous tracking as harness frameworks evolve.