Skip to content

Parallelism Meets Adaptiveness: Scalable Documents Understanding in Multi-Agent LLM Systems

🕒 Published (v1): 2025-07-22 22:42 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A multi-agent LLM framework for SEC 10-K financial document analysis that replaces static pipelines with three adaptive mechanisms: dynamic task routing, bidirectional feedback loops, and Parallel Agent Evaluation (competitive redundancy). Evaluated on SEC filings, it achieves 27% higher compliance accuracy and 74% fewer revision cycles over a static baseline.

Problem

Existing multi-agent LLM frameworks use static role assignments and linear/tree-structured workflows, which fail in high-ambiguity, high-stakes document understanding tasks like financial compliance where evolving context, cross-section inconsistencies, and interpretive nuance demand runtime adaptation and verification.

Method

The framework models task \(T\) as a dependency graph \(G=(V,E)\) over subtasks, coordinated by a central orchestrator agent. Three core mechanisms:

  1. Parallel Agent Evaluation: When a subtask is flagged as ambiguous (confidence below threshold \(\theta\)), \(k\) agents independently generate candidate outputs \(\{o_1,\ldots,o_k\}\). A Critic Agent scores each via a weighted composite: $\(E(o) = w_f \cdot S_{fact} + w_c \cdot S_{coh} + w_r \cdot S_{rel}\)$ with \(w_f=0.5\), \(w_c=0.3\), \(w_r=0.2\); best output \(o^* = \arg\max_i E(o_i)\) is routed downstream.

  2. Dynamic Task Routing: Agents defer subtasks at runtime based on confidence, historical performance scores, and domain markers (e.g., SEC rule references vs. numeric tables).

  3. Bidirectional Feedback Loops: An async feedback bus allows downstream agents to issue revision requests to upstream agents; the orchestrator reassigns or escalates without full workflow re-execution.

A shared long-term memory module persists intermediate outputs and metadata, enabling cross-section reasoning and deduplication.

Key Contributions

  • Parallel Agent Evaluation mechanism: structured competition among agents for high-ambiguity subtasks, with evaluator-driven selection
  • Hierarchical scoring function \(E(o)\) combining factuality (\(S_{fact}\): claim support ratio), coherence (\(S_{coh}\): chain-of-thought critique score), and semantic relevance (\(S_{rel}\): cosine similarity to query)
  • Bidirectional feedback bus enabling targeted upstream revision without full re-runs
  • Modular, plug-and-play architecture (swappable memory backends, domain-specific scoring models, extensible roles)
  • Case study on SEC 10-K filings with ablation over framework components

Results

All metrics averaged over five 10-K filings; Full system vs. Static baseline:

  • Factual coverage: 0.92 vs. 0.71 (+29%)
  • Compliance accuracy: 0.94 vs. 0.74 (+27%)
  • Redundancy penalty: 0.06 vs. 0.22 (−73%)
  • Revision rate: 0.9 vs. 3.4 (−74%)
  • Coherence score (1–5): 4.7 vs. 3.2 (+47%)
  • Relevance score (1–5): 4.9 vs. 3.8 (+29%)
  • Completion time: 115s vs. 134s (−14%)
  • vs. LangGraph supervisor baseline: +14% compliance accuracy in high-ambiguity scenarios
  • Ablation: removing shared memory or feedback loops drops coverage and coherence by >20%

Limitations

  • Parallel evaluation increases inference cost and introduces latency variance; unsuitable for low-latency or cost-constrained deployments
  • Evaluator can introduce systematic bias if scoring criteria misalign with downstream objectives, especially for table-heavy or numeric-heavy sections
  • Evaluated only on SEC 10-K filings; generalizability to weakly structured or less explicitly grounded documents is unvalidated
  • Ambiguity detection and routing policies are heuristic, not learned; learned policies from feedback are deferred to future work
  • No real-time token/cost accounting reported; scalability to very long documents (e.g., 200+ page filings) is unaddressed

Relevance to Harnesses / Meta-Harnesses

This paper is a direct example of a meta-harness pattern: the orchestrator agent acts as a harness that dynamically composes specialist sub-agents, routes tasks based on runtime signals, and applies a centralized evaluator to adjudicate competing outputs — mirroring the judge/selector layer common in meta-harness designs. The Parallel Agent Evaluation mechanism operationalizes the "fan-out + score + select" pattern that many harness designers implement ad hoc, here formalized with an explicit scoring function \(E(o)\). The bidirectional feedback bus and shared memory address the cross-agent state management problem that plagues multi-stage harnesses, providing a concrete architectural solution for propagating corrections without full pipeline reruns. For harness builders, the ablation results quantifying the individual contribution of memory, feedback, and parallel evaluation offer rare empirical guidance on which harness components matter most.