Skip to content

SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding

🕒 Published (v1): 2025-10-30 15:41 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SlideAgent is a two-stage hierarchical multi-agent framework for question answering over multi-page visual documents (slide decks, manuals, financial reports). It pre-builds a query-agnostic knowledge base \(\mathcal{K} = \{K_g, K_p, K_e\}\) at three levels (global, page, element), then selectively activates specialized agents at inference time based on query type. It achieves +7.9% and +9.8% accuracy over proprietary (GPT-4o) and open-source (InternVL3-8B) MLLM baselines on multi-page VQA benchmarks.

Problem

Current MLLMs fail on multi-page visual documents along three axes: (1) they process a limited number of images and reason holistically over pages, missing element-level cues (e.g., miscounting chart segments on cluttered slides); (2) they lack domain-specific visual semantics for financial charts, logos, and spatial conventions; (3) many systems depend on clean metadata (figure locations, hierarchy tags, embedded text) that is unavailable or corrupted in real-world scanned or screenshot PDFs.

Method

Stage 1 — Knowledge Construction (offline, query-agnostic): - Global agent \(M_g\): generates initial document summary \(K_g^{(0)}\) from the first three pages, then performs a single-pass fieldwise rewrite after all page knowledge is collected: \(K_g = M_g^{\text{refine}}(K_g^{(0)}, \text{concat}(K_p^1, \ldots, K_p^{|P|}))\). - Page agent \(M_p\): processes pages sequentially, conditioning each on global knowledge and the previous page: \(K_p^i = M_p(v_i, K_g^{(0)}, K_p^{i-1})\). - Element agent \(M_e\): applies a layout parsing pipeline \(f: v_i \to \{(i, e_j, b_j, t_j)\}\) (text detection + layout detection + element classification) to decompose each page, then generates per-element knowledge \(K_e^j = M_e(i, v_i, e_j, b_j, t_j, K_g, K_p^i)\) including semantic role, functional purpose, and spatial relation to the page.

Stage 2 — Inference (online, query-specific): An agent orchestrator classifies query \(q\) into one of four types (global understanding, fact-based, multi-hop, layout/visual), activating only the relevant agents. Subqueries \(\hat{Q}\) are generated from key entities in \(q\) for retrieval. Retrieved pages \(\hat{P}\) and elements \(\hat{E}\) feed into activated agents to produce partial answers \(h_g, h_p, h_e\). If agents agree, the consensus answer is used directly; otherwise a synthesizer \(\phi\) merges all reasoning traces and retrieved visuals: \(a = \phi(h_g, h_p, h_e, \{v_i : p_i \in R(\hat{P})\})\).

Key Contributions

  • Hierarchical three-level (global/page/element) knowledge construction that is built once and reused across arbitrary queries.
  • Query-type classifier that routes to a minimal agent subset, reducing noise and compute.
  • Subquery generation that expands sparse user queries into entity-targeted retrieval signals, boosting text-retriever performance (SFR: +6.4 MRR, +8.5 nDCG@1).
  • Layout parsing pipeline integrating bounding-box detection with MLLM-generated semantic descriptions, enabling spatial reasoning without document metadata.
  • Global knowledge refinement pass that overwrites early-page bias using full page-level evidence.

Results

  • SlideVQA (proprietary backbone GPT-4o): SlideAgent 84.9% vs. GPT-4o 77.0% (+7.9% overall; +8.3% numeric; +6.5% F1); beats Gemini-2.5-Flash at 83.8%.
  • TechSlides (GPT-4o): +7.5% overall; +12.3% F1 over GPT-4o.
  • FinSlides (GPT-4o): +5.5% overall; +17.5% F1 over GPT-4o.
  • Open-source (InternVL3-8B): +9.8% overall; +11.7% numeric on SlideVQA vs. InternVL3-8B baseline.
  • InfoVQA (single-page): 79.6% vs. ViDoRAG 71.2% (+3.6% overall; +10.5% F1), using InternVL3-8B.
  • Multi-hop reasoning queries: +9.8% (67.4% → 77.2%); visual/layout queries: +7.7% (66.7% → 74.4%).
  • Oracle (ground-truth pages provided): still +7.7% overall and +12.5% numeric on SlideVQA over base model, confirming reasoning gains are independent of retrieval.
  • Retrieval ablation: COLPALI improves +9.7 Recall@3 with SlideAgent artifacts; BM25 gains +4.9 MRR.

Limitations

  • Global agent seeds from only the first three pages; for documents where key context appears late, the initial \(K_g^{(0)}\) prior may be weak despite the refinement pass.
  • Sequential page agent processing is a latency bottleneck for very long documents.
  • 12.5% of failure cases stem directly from OCR errors and tiny visuals — inherent to the layout parsing pipeline.
  • Dominant failure modes (ambiguous questions 22.5%, answer mislocation 17.5%) are retrieval and annotation artifacts not addressable by the agentic design alone.
  • Error analysis based on only 40 annotated failure cases (small sample size).
  • Open-source performance trails Qwen2.5-VL-32B on some splits, indicating strong base models can partially substitute for the harness.

Relevance to Harnesses / Meta-Harnesses

SlideAgent is a prototypical document-processing harness with a fixed two-phase topology: an offline pre-computation stage (three specialized sub-agents building a structured knowledge store) and an online orchestration stage (a query-type router dispatching to a minimal agent subset, followed by a synthesizer merging partial outputs). The query-classification module is a lightweight meta-harness controller — it decides which agents to invoke rather than invoking all unconditionally, a cost-reduction pattern directly applicable to general agentic harness design. The separation of knowledge construction (build once) from inference (query-specific activation) is a reusable harness architecture pattern for any domain where documents or artifacts are queried repeatedly, and the subquery-expansion technique demonstrates how harnesses can improve downstream retrieval without changing retriever internals.