Benchmarking Multi-Agent LLM Architectures for Financial Document Processing: A Comparative Study of Orchestration Patterns, Cost-Accuracy Tradeoffs and Production Scaling Strategies

🕒 Published (v1): 2026-03-24 00:02 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks four multi-agent LLM orchestration architectures—sequential, parallel fan-out, hierarchical supervisor-worker, and reflexive self-correcting—across five frontier and open-weight models on 10,000 SEC filings. Reflexive achieves the highest F1 (0.943) but at 2.3× sequential cost; a combined optimization ("Hierarchical-Optimized") recovers 89% of reflexive accuracy gains at only 1.15× sequential baseline cost. Scaling analysis reveals non-linear accuracy degradation curves with architecture-specific knee points critical for production capacity planning.

Problem

Single-prompt LLM extraction from financial documents faces context-window chunking, hallucination under complexity, and absence of verification. Multi-agent architectures address these but practitioners lack empirical, cost-aware guidance on which orchestration pattern to deploy under different operational constraints (accuracy vs. cost vs. throughput at scale), making architectural decisions consequential and hard to reverse in regulated environments.

Method

Four architectures are implemented in LangGraph v0.2 using a shared set of atomic agents (parser, field extractor, table analyzer, cross-reference resolver, confidence scorer, output formatter):

• Sequential (A): Fixed agent chain; cumulative context passing; \(O(n)\) latency.
• Parallel fan-out (B): Dispatcher routes document sections to domain-specialized extractors concurrently; a merge agent reconciles conflicts via confidence-weighted voting.
• Hierarchical (C): Supervisor maintains a task queue with calibrated confidence threshold (0.85); workers report scores; low-confidence fields are selectively re-extracted (max 2 iterations); supports heterogeneous model routing across workers.
• Reflexive (D): Verifier applies format, cross-field consistency (e.g., \(\text{total assets} = \text{total liabilities} + \text{equity}\)), and source-grounding checks; failures trigger critique-and-correction cycles (max 3 iterations).

Evaluation covers 25 extraction fields across financial metrics, governance, and executive compensation domains, measured on field-level micro-F1 (±2% tolerance for numerical fields), document-level accuracy, p50/p95 latency, cost per document, and token efficiency. Three ablation dimensions are studied: semantic caching (embedding similarity threshold 0.95), model routing (2-tier and 3-tier strategies), and adaptive retry strategies. Scaling from 1K to 100K documents/day on a fixed 8-node cluster surfaces throughput-accuracy degradation curves.

Key Contributions

• Benchmark framework: 4 architectures × 5 models × 10,000 SEC filings = 500 experimental configurations with ground truth from CFA/CPA-credentialed annotators (Cohen's \(\kappa = 0.91\)).
• Empirical Pareto characterization: hierarchical achieves 97.7% of reflexive F1 at 60.7% of its cost.
• Combined optimization (Hierarchical-Optimized): hybrid semantic caching + 2-tier model routing + adaptive retry recovers 89% of reflexive accuracy gains at 1.15× sequential cost ($0.148/doc vs. $0.187 sequential baseline).
• Scaling analysis: architecture-specific knee points (reflexive collapses beyond ~25K docs/day; sequential is most resilient); architectural crossover (hierarchical < parallel F1) at ~75K docs/day on the test cluster.
• Failure taxonomy: 12 failure modes cataloged with architecture-specific prevalence rates and mitigations.

Results

• Best accuracy: Reflexive + Claude 3.5 Sonnet: F1 = 0.943, doc-level strict accuracy = 0.758, $0.430/doc.
• Best cost-accuracy tradeoff: Hierarchical + Claude 3.5 Sonnet: F1 = 0.929, $0.261/doc (60.7% of reflexive cost).
• Combined optimization: Hierarchical-Optimized: F1 = 0.924, $0.148/doc, latency 30.2 s (vs. sequential 0.903 F1, $0.187, 38.7 s).
• Latency: Parallel reduces median latency 1.84× over sequential with +0.014 mean F1 improvement.
• Semantic caching: Field-level caching yields 34.5% cost reduction at −0.005 F1; hybrid adaptive caching reaches 31.4% cost reduction at −0.003 F1.
• Model routing (2-tier Claude + Mixtral): 51.3% cost reduction vs. all-Claude at 98.2% F1 retention.
• Scaling: Reflexive F1 degrades from 0.943 (1K/day) to 0.871 (100K/day); sequential degrades only 0.017 over the same range; reflexive falls below hierarchical at ~50K/day and is worst at 100K/day due to timeout truncation of correction loops.
• Domain difficulty: Financial metrics mean F1 = 0.931 > Governance 0.881 > Executive compensation 0.856 (hierarchical architecture).

Limitations

• Scaling knee points are infrastructure-dependent; the 75K-doc/day crossover will shift with different cluster configurations.
• Dataset limited to U.S. SEC filings (2021–2024); generalization to non-EDGAR, non-English, or private financial documents is untested.
• Ground truth for ~40% of fields relied on manual annotation, not XBRL tags, introducing annotator-dependent noise.
• Models evaluated as of January 2025 snapshots; architecture comparisons may shift as model capabilities change.
• Maximum token budget of 128K for sequential forces section splitting, potentially severing cross-reference dependencies that reflexive/hierarchical architectures handle more gracefully.
• Reflexive architecture cost and latency non-determinism is bounded by a hard 3-iteration cap, which may understate worst-case costs in production.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct empirical study of multi-agent orchestration harness design, providing the kind of quantitative cost-accuracy-throughput tradeoff data that meta-harness builders need to make principled architectural choices. The "Hierarchical-Optimized" configuration—combining semantic caching, heterogeneous model routing, and adaptive retry within a supervisor-worker scaffold—is itself a meta-harness that dynamically selects sub-agents and models at runtime, directly analogous to routing and fallback patterns in production harness frameworks. The scaling analysis and failure taxonomy operationalize harness reliability concerns (queuing under load, timeout truncation, error propagation) that are typically left implicit in framework papers. For practitioners designing general-purpose harnesses, the finding that reflexive (self-correcting loop) architectures degrade fastest at scale while hierarchical supervisor patterns hold Pareto-optimal across most regimes is a concrete design principle transferable beyond the financial domain.