Skip to content

DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations

🕒 Published (v1): 2026-07-22 07:52 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DocOps is a deterministically verifiable benchmark for evaluating LLM agents on end-to-end document manipulation across DOCX, XLSX, PPTX, and PDF formats, using a two-axis taxonomy (3 operation families × 4 difficulty tiers) and artifact-level verifiers that check both requested edits and structural preservation invariants. The central empirical finding is that harness selection—not model scale—is the primary performance driver, with even the best configuration (GPT-5.5 + Codex + skills) achieving only a 0.671 pass rate. Three pervasive failure modes are identified: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata.

Problem

Existing document benchmarks fall into two inadequate paradigms: read-only frameworks (DocBench, DUDE) that treat documents as static knowledge repositories, and workflow-oriented evaluations (OfficeBench, OdysseyBench) that orchestrate application navigation while treating documents as passive data payloads. Neither paradigm treats the document as a first-class computational object requiring stateful, multi-step manipulation with structural validity guarantees. Crucially, prior work uses indirect validation targets (LLM-as-a-judge, round-trip reconstruction) that cannot detect non-destructive edit compliance, and none systematically isolates harness architecture as a controlled experimental variable.

Method

DocOps defines a two-axis taxonomy: the operation axis partitions atomic edits into content (extraction, computation, generation), format (style, highlighting, layout, theme), and structure (reorder, insert/delete, hierarchy, tables); the difficulty axis defines four tiers—L1 (single atomic edit), L2 (multi-dimensional compositional), L3 (single-document workflow), L4 (cross-document workflow).

Task construction follows a four-stage pipeline: (1) seed collection from 600 candidates (472 community requests, 125 public workflows, 3 automation templates), retaining 482; (2) formalization into task_metadata.json with explicit editing scope and preservation requirements; (3) synthetic source-artifact synthesis; (4) iterative PhD-level human review. The final benchmark contains 210 tasks (50 L1, 40 L2, 60 L3, 60 L4), each packaged as a self-contained Harbor bundle with source artifacts, a natural-language instruction, optional document skills, and a deterministic verifier.

Verifiers directly inspect submitted native-format artifacts via document libraries using three predicate types: - Structural predicates: validate executable formulas, outline hierarchies, and object trees - Linguistic anchors: verify required textual content with task-specific keywords - Preservation predicates: check that out-of-scope elements (protected styles, unmodified worksheets) remain intact

Four harnesses are evaluated: DocTools (custom structured document APIs, no shell), Terminus-2 (stateful Bash + file-system access), Claude Code (interactive coding + file-system feedback), and Codex (CLI open script execution). Each is tested with and without official Anthropic document skills (xlsx, docx, pptx, pdf).

All execution is managed by the Harbor framework, which handles container spin-up, agent execution routing, and verifier invocation, providing a uniform interface across heterogeneous harness configurations.

Key Contributions

  • DocOps benchmark: 210 verifiable tasks spanning 4 document formats and 4 complexity tiers, grounded in real-world document-operation patterns
  • Hierarchical taxonomy: two-axis decomposition enabling localization of agent failures to specific operation types and planning depths
  • Deterministic artifact-level verifier: three-predicate design (structural, linguistic, preservation) achieving 95.31% agreement with human audit and 96.67% mutation-detection rate
  • Harness-aware evaluation: first benchmark to treat execution harness as a first-class, systematically controlled experimental variable
  • Failure mode taxonomy: identification and illustration of long-term state tracking collapse, shallow semantic verification, and destructive metadata editing
  • FormatCV metric: coefficient of variation across document formats at each difficulty level, quantifying coupling-induced performance divergence

Results

  • Best configuration: GPT-5.5 + Codex + skills = 0.671 overall pass rate; nearly one-third of tasks fail even at peak performance
  • Harness dominance over model size: switching from DocTools to Terminus-2 typically yields larger gains than scaling model size; DeepSeek-V4-Pro under Terminus-2 (0.467) substantially outperforms Qwen3.5-9B under Codex (0.181) or Claude Code (0.176)
  • DocTools severely underperforms: GPT-5.5 = 0.138, open-source models 0.033–0.114
  • Workflow-level collapse: GPT-5.5 drops from 0.725 (L1) to 0.237 (L4); trend is consistent across all evaluated models
  • Format Ă— difficulty interaction: Excel collapses most sharply (L1: 0.566–0.773 → L3: 0.058–0.085); PDF is most robust at L3 (0.129); \(\text{FormatCV}\) rises from 0.120 at L1 to 0.436 at L3 (non-overlapping 95% bootstrap CIs)
  • Skills asymmetry: explicit procedural skills improve mid-tier models substantially but provide marginal/inconsistent benefit to frontier models (GPT-5.5: 0.648 w/o skill vs. 0.671 w/ skill under Codex)
  • Resource overhead: Claude Code records highest runtime (182.6 s/task with skills) and token usage (664.9K tok/task); DocTools lowest (74.5 s, 31.5K tok)
  • Verifier fidelity: 95.31% agreement with human audit on 128 sampled decisions; 96.67% violation detection across 180 controlled mutations

Limitations

  • Benchmark size is modest (210 tasks), limiting statistical power for fine-grained sub-category comparisons
  • Synthetic task construction—artifacts are purpose-built rather than drawn from naturalistic user corpora—may not fully capture real-world document complexity or ambiguity
  • Verifier false-pass and false-fail rates (3 each out of 128 sampled), while low, are non-zero; complex structural invariants may still escape detection
  • No GUI-based or desktop-automation harnesses are evaluated; the harness comparison is limited to API/CLI/coding interfaces
  • Single primary human reviewer for quality control introduces potential consistency risk
  • Skills tested are Anthropic's official document skills; third-party or custom skill scaffolds are not evaluated, limiting generalizability of the skill-effect findings

Relevance to Harnesses / Meta-Harnesses

DocOps is one of the first benchmarks to treat the execution harness as a first-class experimental variable, systematically comparing four architecturally distinct harnesses (constrained API, Bash-based stateful shell, interactive coding environment, CLI open-script) under controlled model and task conditions—directly quantifying the performance attribution that is otherwise confounded in prior work. The finding that harness architecture explains larger performance gaps than model scale inversion is a strong empirical datum for the harness design research agenda: open, programmable coding environments with file-system feedback (Terminus-2, Claude Code, Codex) consistently dominate constrained RPC-style tool APIs (DocTools) by margins of 3–5× at the same model tier. The Harbor framework functions as a meta-harness, providing the uniform container orchestration and verifier-invocation layer that makes cross-harness comparison reproducible—a direct architectural instantiation of meta-harness principles. Finally, the identification that skill injection disproportionately benefits mid-tier models provides concrete guidance for harness designers deciding when to embed procedural scaffolding versus relying on model-intrinsic planning.