Skip to content

Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows

🕒 Published (v1): 2025-12-15 10:28 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FINCH (FinWorkBench) is an enterprise-grade benchmark of 172 composite finance & accounting workflows sourced from authentic enterprise environments (Enron corpus, financial institutions, governments), spanning spreadsheet manipulation, web search, cross-file retrieval, calculation, and reporting. Even the strongest frontier agent (GPT 5.1 Pro) passes only 38.4% of workflows despite averaging 16.8 minutes per workflow, demonstrating that real-world F&A work remains far from solved.

Problem

Existing benchmarks evaluate agents on isolated, clean table tasks; they do not capture the long-horizon, multimodal, and compositionally messy nature of real enterprise finance & accounting work—where agents must navigate cross-sheet dependencies, irregular layouts, implicit business logic in formulas, versioned spreadsheets, PDFs, and charts across multi-step workflows.

Method

Dataset construction: Three sourcing channels feed into a unified schema—(a) LLM-assisted mining of Enron email threads where collaborators describe workflow goals explicitly, yielding strongly or partially grounded (input + reference) artifact pairs; (b) LLM-based differencing of versioned spreadsheet pairs from Enron/EUSES to infer implicit workflows; (c) curated final-deliverable artifacts from investment firms, World Bank, governments, and existing datasets (WideSearch, DABStep). All instances undergo iterative expert annotation (700+ hours, 5 annotators, ~40% of workflows revised ≥1 round).

Evaluation: Human evaluation assigns binary pass/fail by comparing input, reference, and model output side-by-side. An automated LLM-as-judge pipeline (GPT-5-mini) handles three task types—modify, generate, QA—using structured diffs (diff_ref, diff_model), compact snapshots (first/last 10 rows + first 5 columns + edited cells), and multimodal screenshots. Four rubric dimensions: completeness, correctness, over-edit avoidance, readability.

Agent evaluation: Product-side agents (ChatGPT 5.1 Pro, Claude Sonnet/Opus 4.5) are tested natively with file upload/download; API-based models (GPT 5.1, Claude Sonnet 4.5, Grok 4, Qwen 3 Max, Gemini 3 Pro) use an extended SpreadsheetBench framework with semantic-rich tuple cell encoding (Address, Value, Type, Formula).

Key Contributions

  • 172 enterprise-grade composite workflows with 384 tasks, grounded in 1,710 spreadsheets (27M cells), PDFs, images, and Word documents sourced from real environments
  • Novel workflow construction pipeline combining LLM-assisted email mining, versioned-spreadsheet differencing, and expert annotation
  • Multimodal LLM-as-judge evaluation framework using structured diffs + compact snapshots + screenshots achieving 82–90% agreement with human labels
  • Extended SpreadsheetBench API harness with semantic-rich tuple encoding and tiered PDF/multimodal handling
  • Comprehensive error taxonomy: task misunderstanding (10%), data retrieval errors (25%), formula reasoning errors (35%), code generation errors (25%), rendering errors (5%)

Results

  • GPT 5.1 Pro (product): 38.4% human pass rate (66/172), 41.9% automated; average 16.8 min/workflow; longest failing run ~60 min
  • Claude Opus 4.5 (product): highest pass rate among evaluated systems (exact number not given in excerpt but noted as comparable to GPT 5.1 Pro)
  • Claude Sonnet 4.5 (product): 25.0% human pass rate (43/172), 29.1% automated
  • Long-horizon degradation: GPT 5.1 Pro drops from 44.3% (≤2 tasks) to 23.5% (>2 tasks); Claude Sonnet 4.5 drops from 30.3% to 11.8%
  • Multimodal penalty: GPT 5.1 Pro drops to 35.0% on workflows with non-spreadsheet artifacts (PDF/Word/image)
  • API-based gap: GPT 5.1 via API reaches only 32.0% vs 41.9% for product agent
  • Rubric scores (Sonnet 4.5): completeness 55.8%, correctness 36.0%, over-edit avoidance 55.8%, readability 65.1%
  • LLM-as-judge agreement: 82.1% (GPT 5.1 Pro) and 90.2% (Claude Sonnet 4.5) accuracy vs human labels; judge tends to overestimate by a few percentage points

Limitations

  • Only 172 workflows; single-call API baselines preclude iterative self-correction, understating API model potential
  • LLM-as-judge (GPT-5-mini) can miss subtle visual/numerical errors in large spreadsheets or be overly literal; automated scores are approximate
  • Spreadsheet tooling pipeline has incomplete support for corrupted/uncommon file formats, causing false failures
  • Workflow sourcing is heavily Enron-centric; geographic and industry diversity is limited (though supplemented by government/financial institution artifacts)
  • 78.5% multi-task workflows bias evaluation toward long-horizon capability; isolated task-level performance is not independently benchmarked
  • GPT-5-mini assigns all four rubric scores in a single call, introducing potential inter-rubric confusion

Relevance to Agentic AI / LLM Agents

FINCH is a direct stress test of multi-step agent capabilities in a high-stakes, document-rich professional domain, making it directly relevant to researchers studying tool-using agents, long-horizon planning, and error accumulation in agentic pipelines. The sharp pass-rate drop from ≤2 to >2 tasks quantifies compounding failure—a fundamental challenge for sequential LLM agents—while the error taxonomy (formula reasoning, cross-sheet retrieval, code generation) pinpoints concrete bottlenecks. The LLM-as-judge pipeline extended with structured diffs and multimodal rendering is itself a methodological contribution for scalable agentic evaluation. For the broader agentic AI research community, FINCH establishes that product-grade agents with rich interactive affordances still fall below 50% on realistic enterprise workflows, setting a clear capability target.