SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows

🕒 Published (v1): 2026-06-29 08:33 UTC · Source: Arxiv · link

Why this paper was selected

End-to-end agent benchmark; benchmarks are core harness infrastructure

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A workflow-level benchmark (321 tasks, avg 11.8 sheets, 593.5 cell modifications per task) for spreadsheet agents covering generation, debugging, and visualization on authentic business data. The best model (Claude Opus 4.6) achieves only 34.89% overall accuracy; debugging accuracy bottoms out at 12.00%, highlighting a large gap between current agents and real-world spreadsheet automation requirements.

Problem

Existing spreadsheet benchmarks (SheetCopilot, InstructExcel, SpreadsheetBench) evaluate isolated operations — single-formula generation or local cell edits — using synthetic or forum-derived data with simple 1–2 sheet workbooks. They fail to capture end-to-end business workflows that require cross-sheet reasoning, long-range dependencies, multi-step coordination, and domain-specific financial knowledge. No prior benchmark provides a systematic error taxonomy for debugging or native chart-generation evaluation within the spreadsheet paradigm.

Method

SpreadsheetBench 2 is constructed through a three-stage pipeline: (1) data collection from authentic business sources (NYU Stern/Damodaran, Screener.in, Bloomberg, BSE India) curated by financial experts into gold-standard spreadsheets; (2) task construction by transforming complete workbooks into partially-specified inputs paired with natural-language instructions (selective region removal, controlled error injection, visualization objectives); (3) dual-expert validation with independent blind solves and discrepancy resolution.

The benchmark covers three task categories mapped to workflow stages: - Generation — Financial Modeling (avg 15.3 sheets, 1,164.5 modified cells) and Template (avg 1.2 sheets, 65.7 cells) - Debugging (avg 21.1 sheets, 656.6 cells) — 10 error types (double counting, incorrect references, formula errors like #REF!/#NUM!, etc.) - Visualization (avg 1.7 sheets) — chart construction from irregularly-structured multi-sheet data

Evaluation uses two protocols: Modification (fraction of target cells matching golden file) and Accuracy (binary task-level all-cells-match), plus a VLM-as-a-judge rubric (GLM-4.6V) for visualization tasks. The agent scaffold is built on SWE-agent with three tools (bash, view_xlsx, submit), max 50 turns, fixed across models.

Key Contributions

• First workflow-level spreadsheet benchmark shifting evaluation from isolated cell edits to end-to-end task completion over complex multi-sheet workbooks with cross-sheet dependencies
• Systematic evaluation of 8 frontier LLMs (3 closed-source, 5 open-source) under a unified multi-turn agent scaffold, plus 4 LLM-based spreadsheet products
• Construction from authentic business data with 1,500+ person-hours of expert annotation and dual-expert validation
• A 10-category error taxonomy for debugging tasks and a 6-type failure taxonomy from trajectory analysis
• Scaffold-design ablation fixing the model (GLM-5) across four scaffolds, showing SWE-agent scaffold outperforms general-purpose coding agent scaffolds on end-to-end correctness

Results

• Best model (Claude Opus 4.6): 34.89% overall accuracy; debugging accuracy only 12.00%
• Closed-source models dominate: top three (Claude Opus 4.6, GPT-5.2, Gemini 3.1 Pro) score 23.68%–34.89% vs open-source (7.17%–17.14%)
• Task difficulty ordering: Visualization (62.5% best accuracy) → Template (52.58%) → Financial Modeling (34.00%) → Debugging (12.00%)
• High Modification but low Accuracy gap: Claude Opus 4.6 achieves 89.69% Modification on Financial Modeling but only 34.00% Accuracy — models do cell edits ok but fail at cross-cell consistency
• LLM-based spreadsheet products underperform: best (Claude for Excel) achieves only 15.4% vs Claude Opus 4.6's 45.2% on the 30-sample subset
• Failure taxonomy (Claude Opus 4.6 trajectories): dominant bottlenecks are insufficient spreadsheet inspection and incorrect target-cell selection
• Scaffold comparison (GLM-5, 50 samples): SWE-agent scaffold outperforms Claude Code, Kilo Code, and Cline on task-level accuracy despite comparable modification scores
• Error type difficulty: "Incorrect Index Match" and "Incorrect Sign" are most tractable; "Errors" (#REF!, #NUM!) is hardest (<20% even for best model)
• Claude Opus 4.6 maintains higher effective-step ratio (fewer tool-call/code-execution failures) than MiniMax M2.5, requiring fewer interaction steps for better results

Limitations

• Domain coverage is finance-heavy (financial reports, DCF, LBO, M&A) — generalizability to other spreadsheet domains (scientific, operational, inventory) is untested
• Visualization evaluation depends on a VLM judge (GLM-4.6V) whose own accuracy ceiling is unknown — rubric pass rates may conflate agent and judge errors
• The SWE-agent scaffold, while fixed across models, was itself the best-performing scaffold — results may shift with alternative scaffold designs
• 321 tasks is relatively small compared to NLP benchmarks; fine-grained per-category or per-error-type statistical power is limited
• Only English-language instructions and Western financial reporting conventions are used

Relevance to Harnesses / Meta-Harnesses

SpreadsheetBench 2 is a domain-specific agent evaluation harness that demonstrates a key meta-harness principle: evaluation at the workflow level reveals bottlenecks invisible to isolated-operation benchmarks. The large Modification-vs-Accuracy gap shows that cell-level proxy metrics overestimate agent competence — a warning for harness design in other domains. The scaffold ablation (SWE-agent vs coding agents) also contributes to the meta-harness literature by quantifying how scaffold design directly shapes end-to-end task performance, independent of the underlying model.