Skip to content

FinSAgent: Corpus-Aligned Multi-Agent RAG Framework for Evidence-Grounded SEC Filing Question Answering

🕒 Published (v1): 2026-07-20 16:03 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FinSAgent is a multi-agent RAG framework for SEC filing question answering that addresses prior–corpus misalignment: the mismatch between an LLM's internalized priors and the target corpus's structure, terminology, and evidence standards. It combines role-specialized parallel agents, database-aware query decomposition, and a learned feature-gated reranker to improve both retrieval coverage and answer correctness. Across five financial QA benchmarks and an online user study, it outperforms single-agent and multi-agent baselines under matched evidence budgets.

Problem

Existing RAG and multi-agent systems generate retrieval queries purely from model priors and rank candidates by semantic similarity. For SEC filings this produces two failure modes: (1) front-end: sub-queries miss corpus-specific terminology and structure, so disclosed evidence is never retrieved; (2) back-end: semantic rerankers surface topically similar but evidentially invalid chunks (e.g., near-verbatim boilerplate recurring across issuers), crowding out the true disclosure. Neither problem is addressed by existing query decomposition or multi-agent debate approaches.

Method

FinSAgent is a three-stage pipeline orchestrated by a central routing agent:

  1. Role-specialized parallel multi-agent analysis (Stage 1). Five agents \(\mathcal{A} = \{a_g, a_q, a_m, a_l, a_c\}\) (general, quantitative, market, legal, company) operate in parallel. Each carries a role policy \(\pi_a = (\rho_a, \Omega_a, \phi_a)\) and produces a role-conditioned analytical view \(z_a = g_a(q; \pi_a)\) of the question, anchored to the mandated 10-K item structure.

  2. Database-aware query decomposition (Stage 2). A lightweight FAISS-based retriever maps \(q\) to the top-\(k_0\) section-level summaries \(R_{k_0}^{sv}(q)\) of the local corpus. Each agent decomposes the question into role-specific sub-queries conditioned on both its analytical view and this corpus view: \(Q_a = d_a(q, z_a, R_{k_0}^{sv}(q))\). This exposes corpus-specific terminology and section headings at decomposition time.

  3. Multi-path retrieval with feature-gated reranking (Stage 3). Candidates per sub-query are the union of BM25 sparse (\(R_k^{sp}\)), dense embedding (\(R_k^{de}\)), and section-summary (\(R_k^{sm}\)) paths. A cross-encoder produces a base score \(s^{rr}\); a LightGBM model \(M\) trained on 31 non-semantic features (path scores, rank provenance, query/chunk metadata, lexical overlap) produces a validity estimate \(\hat{p}\). The gated score is: $\(\tilde{s}(c, q_{a,j}) = s^{rr}(c, q_{a,j})\bigl(1 - \lambda_a(1 - \hat{p}(c, q_{a,j}))\bigr)\)$ where \(\lambda_a \in [0,1]\) is a per-agent penalty strength. The multiplicative form penalizes high-confidence false positives most heavily. The gate is trained with hard-negative mining on an in-distribution split.

Key Contributions

  • Frames SEC filing QA as a corpus-alignment problem at both retrieval planning and evidence selection stages; introduces prior–corpus misalignment as a diagnostic lens.
  • Role-specialized parallel multi-agent module anchored to the mandated 10-K item structure, with every role shown to retrieve exclusive ground-truth chunks.
  • Database-aware query decomposition: query-time FAISS lookup of section-level summaries injects corpus-specific terminology into sub-query generation without encoding the full corpus in the prompt.
  • Feature-gated reranker (LightGBM over 31 non-semantic features) separating evidential validity from semantic similarity via a multiplicative gate; includes hard-negative training and SHAP attribution.
  • Evaluation on five benchmarks (FinanceBench, FinDER, SECQUE, Lotus, Zeekr) plus a three-arm online RCT with ~1,400 anonymous user ratings.

Results

  • FinSAgent (Qwen3-Max backbone) achieves the highest Correctness on all five benchmarks vs. Naive RAG, Naive RAG+BM25, FinSage, FinGPT, MoA, and FinDebate.
  • Largest gains on the Factual Consistency dimension across datasets, confirming evidence-grounding improvement.
  • Ablation (Table 3, fixed ~35-chunk budget): multi-role retrieval drives large Macro-Recall gains; feature-gated reranking is most impactful on the hard Lotus set; database-aware decomposition peaks FinanceBench at Macro-Recall 80; full system achieves best overall coverage.
  • Matched-budget fairness study: with MoA and FinDebate given FinSAgent's per-role budget but with FinSAgent's two mechanisms disabled, FinSAgent still wins Factual Consistency on all five benchmarks and Correctness on four of five.
  • Online RCT (~1,400 anonymous ratings): FinSAgent receives higher scores than baselines.
  • Full system dips on FinanceBench retrieval (80 → 76 Macro-Recall) when operating under the fixed chunk budget, attributed to routing overhead under budget constraint.
  • Gains are backbone-agnostic: consistent across DeepSeek-v3.1-Terminus, Kimi-K2.5, and Qwen3-Max.

Limitations

  • Evaluated exclusively on SEC 10-K/10-Q filings; generalization to other structured document corpora (e.g., earnings call transcripts, regulatory filings outside the U.S.) is unverified.
  • The LightGBM gating model requires an in-distribution labeled training split for hard-negative mining; acquiring such labels for a new domain adds overhead.
  • The five fixed agent roles are derived from 10-K structure and may not cover evidence categories in other filing types or jurisdictions.
  • Fixed ~35-chunk budget creates a tension between coverage and routing: under budget pressure, the full system can underperform configurations with simpler routing on individual benchmarks (FinanceBench 80 → 76 dip).
  • The section-level summary index requires offline preprocessing; freshness and scalability for real-time filing ingestion are not discussed.
  • The online RCT uses anonymous users whose financial expertise is uncharacterized; ratings may not reflect analyst-grade evidence demands.

Relevance to Harnesses / Meta-Harnesses

FinSAgent is a concrete implementation of an orchestrator-agent harness pattern: a central routing orchestrator dispatches to a pool of role-specialized subagents, coordinates their parallel execution, aggregates sub-answers, and synthesizes a final response — matching the canonical "meta-harness" architecture where a controller manages a dynamic agent fleet. The paper's database-aware decomposition mechanism is directly analogous to harness-level context injection, where the orchestrator enriches each subagent's prompt with retrieved corpus state before dispatch rather than relying solely on the original query. The matched-budget ablation provides a methodologically clean contribution to harness design: it isolates the value of the orchestration mechanisms (corpus-aware routing and evidence gating) from the value of simply running more agents, a confound that harness evaluations frequently fail to control. The feature-gated reranker adds a learned validity layer on top of semantic ranking, representing a pattern for harnesses that must filter agent outputs for evidential reliability rather than topical relevance.