Skip to content

FinanceComplexQA: Benchmarking Agentic Reasoning on Industrial-grade Financial Documents

🕒 Published (v1): 2026-07-21 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FinanceComplexQA is a bilingual (EN/CN) benchmark of 2,026 expert-level open-ended QA tasks over 1,009 industrial-grade financial documents, requiring dual-context reasoning (explicit document evidence + implicit domain knowledge) and cross-layout evidence fusion. Alongside the benchmark, the authors release Finance-LaTeX SKILL, a multi-agent synthesis pipeline that generates 2,000 professional financial documents and 6,000 QA pairs with layout-aware LaTeX structure. Evaluation of RAG, MCP, and agentic systems shows a large performance gap between current systems and professional financial analyst capability.

Problem

Existing financial QA benchmarks target narrow subtasks—table QA, numerical extraction, short-context retrieval, or summarization—using synthetic or template-based questions that do not capture the multi-step analytical reasoning required in real financial research. They rarely require joint reasoning over heterogeneous document elements (paragraphs, tables, forms, charts), lack stable open-ended reference answers, and have no consistent evaluation protocol for analytical responses. No prior benchmark simultaneously covers bilingual expert questions, long PDF-parsed documents, cross-layout reasoning, and open-ended generation.

Method

Finance-LaTeX SKILL is a multi-agent framework operating in four stages: (1) evidence planning—an agent samples financial domain, audience, terminology constraints, and classifies evidence units by reasoning function (direct lookup, numerical comparison, multi-hop inference, etc.); (2) layout-aware LaTeX generation—documents are written as realistic financial reports with sections, tables (row/column labels, units, periods), formulas, captions, and adjacent explanatory text, creating intentional cross-layout dependencies; (3) QA pair generation—each question is paired with required evidence units, a reference answer, and reasoning notes; the generator rejects questions answerable by span-copying, dependent on volatile data, or unsupported by speculation; (4) multi-channel verification—LLM coherence checks, web/MCP-tool fact verification, and harness tests for evidence-answer reproducibility, with a check-and-correct loop.

FinanceComplexQA is built from real financial corpora (corporate reports, investment strategy reports, bank statements, compliance audits, government bulletins, etc.), parsed into layout-aware evidence hierarchies (page-local and document-level topic blocks). Each question requires the system to combine explicit document evidence with implicit domain knowledge (margin interpretation, debt pressure, sector cyclicality, accounting relations). The 9 task types span comparison, implicit reasoning, knowledge query, multi-hop reasoning, summarization, planning, explicit reasoning, multi-hop judgment, and numerical comparison. Agent-as-a-Judge evaluation assigns task-specific scores across six dimensions: Accuracy (ACC), ROUGE overlap (ROU), Faithfulness (FS), Coverage (Cov), numeric correctness, and cost/latency.

Four systems are compared: LightRAG (flat retrieval baseline), PageIndex (layout-aware retrieval), and two agentic systems (Codex-style and Claude Code-style) across 4 closed-source and 3 open-source models.

Key Contributions

  • Finance-LaTeX SKILL: multi-agent pipeline for scalable synthesis of layout-rich financial documents and QA pairs (2,000 docs, 6,000 QA pairs), with multi-channel verification and evidence-planning.
  • FinanceComplexQA benchmark: 2,026 expert-level questions over 1,009 real financial documents; bilingual; 8 financial scenarios; 9 task types; average document length of ~16,493 tokens (English docs average ~24,112 tokens, Chinese ~12,040 tokens).
  • Dual-context reasoning paradigm: benchmark design requiring both explicit document retrieval and implicit domain knowledge for every question.
  • Error taxonomy: schema distinguishes retrieval failures (missed table/chart/footnote) from reasoning failures (correct evidence, wrong conclusion), enabling targeted system improvement.
  • Systematic evaluation of RAG, MCP, and agentic systems, with detailed failure-mode analysis across numerical computation, multi-hop reasoning, summarization, and industry analysis.

Results

  • Claude Code + Sonnet 5 achieves the best overall scene scores: 76.01 (Chinese) and 69.39 (English).
  • Codex + GPT-5.5 achieves the strongest aggregate task averages among closed-source settings: 60.63 (Chinese) and 62.97 (English).
  • Codex + DeepSeek-V4-Flash is the strongest open-source configuration: 60.44 (Chinese) and 61.55 (English).
  • Layout-aware retrieval (PageIndex) consistently outperforms flat retrieval (LightRAG) across scenarios, highlighting that preserving page structure and table context is critical.
  • Agentic systems generally outperform RAG baselines overall, but no single system dominates all 9 task types or all 8 scenarios; PageIndex is especially competitive on knowledge query, multi-hop reasoning, and numerical tasks.
  • All systems fall substantially short of the benchmark's implicit professional-analyst standard; long-chain numerical reasoning and cross-layout evidence fusion are the primary bottlenecks.

Limitations

  • The paper is partially truncated; not all ablation or per-task breakdown numbers are available in the provided text.
  • Agent-as-a-Judge evaluation introduces potential judge model bias; judge accuracy on financial reasoning is not independently validated.
  • Fixed reference analytical answers may penalize legitimate alternative reasoning chains.
  • Benchmark relies on the Finance-LaTeX pipeline for development/ablation data; pipeline-generated documents may share systematic artifacts.
  • English and Chinese document lengths differ substantially (avg. ~24k vs. ~12k tokens), making direct cross-lingual comparison confounded by length.
  • Only 4 pipeline configurations are evaluated; broader coverage of retrieval and agent strategies is absent.
  • Task labels are explicitly non-exclusive cognitive categories, making fine-grained capability attribution difficult.

Relevance to Agentic AI / LLM Agents

FinanceComplexQA directly stress-tests the agentic reasoning stack—tool invocation, multi-stage planning, evidence fusion, and answer revision—in a domain where errors have real-world consequences, making it a harder and more realistic probe than existing benchmarks. The finding that orchestration strategy (RAG vs. MCP vs. agentic loop) and core model jointly determine performance is actionable for agentic system design. The Agent-as-a-Judge evaluation protocol is itself a contribution to agentic evaluation methodology, offering a reusable scaffold for open-ended analytical answer scoring. The error taxonomy (retrieval failure vs. reasoning failure) provides a diagnostic lens applicable to any document-grounded agent pipeline.