Skip to content

Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction

🕒 Published (v1): 2026-07-23 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Tencent WorkBuddy Bench is a 260-task, four-domain coding-agent benchmark (Code, Web, Office, Security) built around a shared, open, Harbor-style task-directory harness and run reproducibly on two agent backends. Contamination resistance comes from reverse-engineering real commits and CVEs into colloquial, role-played prompts whose text is not web-searchable, backed by dataset versioning rather than secrecy. No suite-wide average is reported because the four subsets use incompatible scoring instruments.

Problem

Existing agent benchmarks fail on one of two axes: static public suites (SWE-bench, VisualWebBench) leak via web-searchable prompts and cover narrow task types; vendor production benchmarks (CursorBench) have task distributions grounded in real usage but are closed and unauditable. No prior suite simultaneously covers multi-domain real-work tasks (repository SWE, front-end, office workflows, red/blue security), resists prompt contamination by construction, and is fully reproducible and open.

Method

Contamination-resistant construction. Every task is reverse-engineered from a real upstream artifact (OSS commit, pull request, CVE) or a concrete business scenario, then rewritten as a short, colloquial, underspecified natural-language request voiced through a domain-appropriate persona. The rewrite withholds the root cause, reference diff, and any framing recoverable by web search. Resistance rests on the rewrite, not on secrecy; versioning and optional canary strings manage post-release exposure.

Unified task-directory format (Harbor-style delta). Each task is packaged as: 

tasks/<task-name>/
  task.toml          # metadata: category, difficulty, timeouts
  instruction.md     # agent-visible NL request
  environment/
    Dockerfile       # builds only the visible workspace
  workspace/
  tests/
    test.sh          # post-episode entry point
    grading/         # scoring assets (hidden at solve time, public at release)
The Dockerfile builds only workspace/; tests/ stays outside the image, so the evaluation boundary is a packaging property, not runtime configuration.

Admission gate. Each candidate task clears a two-run oracle check: baseline reward \(\leq 0.3\) (workspace does not already satisfy the contract) and oracle reward \(= 1.0\) (at least one reference solution achieves full verifier reward).

Per-track scoring. Code uses hidden test suites; Web uses a three-layer rubric (deterministic rule checks + LLM/VLM judges + an agent-judge that drives the running artifact for interactive-state verification); Office uses a task-specific blend of deterministic rule checks and evidence-grounded LLM judge rubrics with preconfigured weights; Security uses a fully deterministic scoring.py. Scores are not aggregated across tracks.

Dual-harness execution. All 260 tasks run under both CodeBuddy Code and Claude Code in isolated, no-internet sandboxes, enabling cross-harness and cross-model comparison.

Key Contributions

  • A 260-task, four-domain open benchmark (Code 80 / Web 70 / Office 50 / Security 60) with a common task-directory schema and execution protocol
  • Contamination-resistant construction via persona-rewritten, colloquial reverse-engineering of real commits, PRs, and CVEs — not reproductions of public issue text
  • A multi-instrument evaluation methodology: oracle-gated admission, hidden-test scoring, LLM/VLM/agent-judge rubrics, and fully deterministic security scoring, all with post-episode evaluation isolation
  • A cross-model leaderboard under two agent harnesses across multiple model families

Results

All figures from the cross-model leaderboard (CodeBuddy Code + Claude Code, per-subset scores reported separately; no suite-wide average by design; "Overall Score" appears to be a weighted combination):

  • Claude Opus 4.8: Code 74.4%, Web 68.1%, Office 82.4%, Security 64.4%, Overall 75.0% (rank 1)
  • GLM-5.2: Code 71.5%, Web 67.4%, Office 79.6%, Security 76.3%, Overall 72.9% (rank 2)
  • GPT-5.5: Code 72.9%, Web 61.1%, Office 82.0%, Security 64.4%, Overall 72.0% (rank 3)
  • HY-3: Code 62.9%, Web 67.7%, Office 82.1%, Security 64.5%, Overall 70.9% (rank 4)
  • Kimi K2.7: Code 63.4%, Web 54.3%, Office 80.9%, Security —, Overall 66.2% (rank 5)
  • MiniMax-M3: Code 60.1%, Web 58.0%, Office 78.3%, Security 74.1%, Overall 65.5% (rank 6)
  • DeepSeek-V4-Pro: Code 58.9%, Web 54.6%, Office 79.1%, Security 70.0%, Overall 64.2% (rank 7)

Observed failure modes in Code: agents looping on test-file edits until timeout, and agents navigating to wrong files in large codebases — indicating difficulty stems primarily from cross-module navigation rather than code synthesis.

Limitations

  • Contamination resistance is construction-level only: models may have already seen underlying OSS commit code or CVE analyses, and post-release training exposure is mitigated by versioning but not eliminated
  • 260 tasks total is modest; per-track counts (50–80) limit statistical power for fine-grained difficulty analysis
  • Version numbering semantics differ across subsets; version numbers are not directly comparable cross-track
  • No suite-wide aggregate score is possible given incommensurable scoring instruments — cross-subset comparison requires per-track reading
  • Security subset scoring is entirely deterministic (no semantic rubric), which may miss partial credit for nuanced analyst outputs
  • The "Overall Score" computation method is not fully specified in the provided text

Relevance to Harnesses / Meta-Harnesses

WorkBuddy Bench is itself a meta-harness: it defines a common task-directory schema (derived from Harbor) and a shared execution lifecycle that wraps four heterogeneous domain-specific scoring harnesses under one reproducible protocol, making it a direct case study in how to build a harness that unifies disparate evaluation backends. The two-layer architecture — a packaging/execution meta-harness above track-specific scoring instruments — is a reusable pattern for anyone designing multi-domain agent evaluation infrastructure. The oracle-gated admission protocol (baseline \(\leq 0.3\), oracle \(= 1.0\)) is a concrete quality gate that could be adopted in other harness designs. Running all tasks on two named agent harnesses (CodeBuddy Code, Claude Code) and releasing the full harness code also makes it a reference implementation for reproducible, cross-harness agent evaluation.