Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

🕒 Published (v1): 2026-05-05 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Workspace-Bench: AI agents on real heterogeneous file-dependency workspace tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Workspace-Bench is a benchmark for evaluating AI agents on Workspace Learning — the ability to identify, reason over, and update explicit and implicit dependencies among heterogeneous files in a worker's digital workspace. It provides 5 realistic worker-profile environments, 74 file types, 20,476 files (up to 20 GB), and 388 tasks with 7,399 rubrics. Across 27 configurations of 4 agent harnesses and 7 foundation models, the best agent reaches only ~60%, far below the human+tool ceiling of 80.7%.

Problem

Existing agent benchmarks fall into four categories — prompt-only, tool/environment-driven, task-file-driven, and workspace-relevant — none of which adequately stress-test inter-file relational reasoning. Prior workspace-relevant benchmarks (OfficeBench, TheAgentCompany, SWE-Bench) are limited to monolithic file-system structures with fewer than 20 file modalities and entirely omit file lineage relations (version history, derivation chains). They also treat multi-file involvement as an implicit byproduct rather than an explicitly evaluated capability axis, making them blind to the core bottleneck of real office workflows.

Method

Workspace construction follows a two-stage, top-down hybrid pipeline: 1. Structure generation: persona-conditioned agents generate role-specific directory hierarchies (e.g., src/, experiments/, results/) with injected structural noise (redundant folders, ambiguous names, archive directories). 2. Content population: a semantic-driven agentic crawler retrieves real public resources (arXiv papers, GitHub repos, spreadsheets, reports) matched to directory semantics; LLMs then synthesize derivative artifacts (emails referencing papers, meeting notes linked to design docs).

Task curation: 388 tasks are manually annotated from real office scenarios on the Lark platform (ByteDance); each task is intentionally under-specified so agents must recover file dependencies. Each task is paired with a file dependency graph annotating five relation types: link, embed, semantic, lineage, and derivative.

Evaluation framework: Six capability dimensions — Workspace Comprehension, Heterogeneous File Understanding, Task-Supporting File Search, Result-Providing File Aggregation, Inter-File Content Relation Capturing, and Inter-File Lineage Relation Tracing — are scored via 7,399 rubrics (avg. 19.1 per task) covering both final outputs and intermediate decisions. Evaluation runs in sandboxed parallel execution with automatic workspace recovery. A 100-task Workspace-Bench-Lite subset preserves distribution while cutting evaluation cost by ~70%.

Evaluation scope: 4 agent harnesses (DeepAgent, Hermes, OpenClaw, and one additional) × 7 backbone LLMs (GLM-5.1, Qwen, MiniMax-M2, GPT-5, Gemini, and others) = 27 evaluated configurations.

Key Contributions

• Workspace-Bench: 5 realistic worker-profile workspaces with 20,476 files across 74 file types, 388 dependency-annotated tasks, and 7,399 rubrics covering 6 evaluation dimensions; the only benchmark with explicit file lineage relation evaluation.
• Workspace-Bench-Lite: 100-task stratified subset reducing evaluation cost by ~70% while preserving benchmark distribution.
• Evaluation framework: sandboxed parallel execution + automatic workspace recovery + dependency-graph-based rubric scoring, enabling fine-grained assessment of intermediate reasoning steps, not only final outputs.
• Empirical baseline: systematic evaluation of 27 agent-harness × LLM configurations, revealing concrete failure modes (cost explosion, lineage blindness, harness-model interaction effects).

Results

• Best single configuration (DeepAgent + GLM-5.1): ~60% rubrics pass rate
• Average across all 27 configurations: 43.3%
• Human + tools: 80.7% (gap of ~37 pp to the best agent)
• Performance degrades monotonically with task difficulty: Easy 57.6% → Hard 40.5%
• DeepAgent + MiniMax-M2.7 exhibits cost explosion: up to 58.1 interaction turns and 0.61 M tokens per task, yet achieves only ~45% pass rate
• Harnesses provide limited lift on powerful backbone LLMs but measurably boost weaker models in dependency-aware tasks
• Heterogeneous file understanding and lineage tracing are the primary bottlenecks across all configurations

Limitations

• Workspaces are partially LLM-synthesized rather than extracted from actual private enterprise file systems, which may underrepresent proprietary formats and organizational conventions not captured by public crawl data.
• Paper text is truncated; explicit discussion of benchmark coverage gaps, adversarial robustness, and longitudinal workspace dynamics is not available in the provided excerpt.
• Evaluation rubrics are human-curated but rubric coverage is finite (~19 per task); an agent could satisfy rubrics while still failing qualitatively.
• The benchmark is static — it does not model concurrent multi-user workspace evolution or real-time file mutations.
• Only 4 harnesses are evaluated; coverage of newer or proprietary harness architectures is absent.

Relevance to Harnesses / Meta-Harnesses

Workspace-Bench is one of the first benchmarks to treat the harness as a first-class experimental variable, directly measuring how harness choice interacts with backbone LLM capability on realistic multi-file tasks. The finding that harnesses amplify weaker models but plateau for strong ones has immediate implications for harness design: the marginal value of orchestration logic diminishes as the underlying model improves. The documented "cost explosion" — where an open-source harness consumes 0.61 M tokens per task and 58 turns while still underperforming — provides a concrete, reproducible failure mode that meta-harness designers can target. For researchers building evaluation harnesses for agentic systems, the sandboxed-execution + dependency-graph-rubric architecture is a directly transferable design pattern for process-aware, intermediate-decision scoring.