Skip to content

DocLens : A Tool-Augmented Multi-Agent Framework for Long Visual Document Understanding

🕒 Published (v1): 2025-11-14 18:42 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DocLens is a tool-augmented multi-agent framework for question answering over long visual documents that decomposes the task into a two-stage pipeline: a Lens Module for evidence localization (page + element level) and a Reasoning Module with sampling-adjudication for reliable answer generation. Paired with Gemini-2.5-Pro, it achieves 67.6 on MMLongBench-Doc, surpassing the human expert baseline of 65.8 for the first time. The framework also substantially reduces hallucination on unanswerable queries.

Problem

Existing VLMs and retrieval-based methods fail at two levels of evidence localization in long visual documents: (1) page-level recall is poor—Gemini-2.5-Pro alone recalls only 68% of evidence pages and vector-based methods (ColBERT+ColPali) achieve 55.3% Recall@10—and (2) even on correct pages, fine-grained visual elements (charts, tables, figures) are too small or dense to interpret at full-page resolution. This dual failure causes hallucination on over half of unanswerable queries.

Method

DocLens decomposes the answer function \(A = f_{\text{generate}}(f_{\text{extract}}(D, Q), Q)\) into two orchestrated modules:

Lens Module (\(f_{\text{extract}}\)): - Page Navigator: OCR-augments every page, then prompts an LLM with interleaved screenshots and OCR text \(T_e\) times at temperature \(\tau > 0\); the predicted evidence set is the union \(\mathcal{E}_{\text{pred}} = \bigcup_{j=1}^{T_e} \mathcal{E}^{(j)}\). Pages are chunked and processed in parallel when context limits are hit. - Element Localizer: For each retrieved page \(P_k \in \mathcal{E}_{\text{pred}}\), runs layout detection to obtain bounding boxes, then crops visual elements \(\mathcal{V}_k = \{\text{Crop}(P_k, \text{bbox}) \mid \text{bbox} \in \text{LayoutDetect}(P_k)\}\), assembling a rich evidence set \(\mathcal{S} = \{(P_k, T_k, \mathcal{V}_k)\}\).

Reasoning Module (\(f_{\text{generate}}\)): - Answer Sampler: Generates \(T_a\) diverse chain-of-thought reasoning-answer pairs \(\{R_i, A_i\}_{i=1}^{T_a}\) at temperature \(\tau > 0\). - Adjudicator: Cross-validates all candidates and selects the final answer \(A_{\text{final}} = \text{LLM}_{\text{Adjud}}(\{(R_i, A_i)\}_{i=1}^{T_a})\).

A hybrid-backbone variant uses a cheaper model (Gemini-2.5-Flash or Flash-Lite) for the Page Navigator and an expensive model for reasoning.

Key Contributions

  • Lens Module: OCR-augmented, sampling-based Page Navigator achieving 97.3% evidence page recall (vs. 71.1% for vector-based MDocAgent and 89.0% for SimpleDoc), within 1.5% of oracle-page performance.
  • Element Localizer: Layout-detection-and-crop pipeline that improves block-level evidence identification by +4.9% precision, +9.3% recall, +6.7% F1 on FinRAGBench-V visual citation subset.
  • Sampling-Adjudication mechanism: Reduces hallucination on unanswerable queries by +8.2–13.8% absolute across backbone models.
  • State-of-the-art results: First system to surpass human experts (65.8) on MMLongBench-Doc, reaching 67.6 with Gemini-2.5-Pro.
  • Hybrid backbone: Using Gemini-2.5-Flash-Lite for retrieval (cost $0.1/M tokens vs. $1.25/M for Pro) still outperforms the vanilla Gemini-2.5-Pro baseline (64.4% vs. 63.3%).

Results

  • MMLongBench-Doc (human expert = 65.8):
  • DocLens + Gemini-2.5-Pro: 67.6 (surpasses human); +4.6 over OCR-augmented baseline (63.0 implied) and +4.4 over best prior agentic framework SimpleDoc (67.5 for Pro in table)
  • DocLens + Gemini-2.5-Flash: 64.7; +13.0% absolute on Unanswerable subset
  • DocLens + Claude-4-Sonnet: 63.3; +8.2% absolute on Unanswerable
  • FinRAGBench-V:
  • DocLens + Gemini-2.5-Pro: 70.4 (+5.5% over best baseline)
  • DocLens + Gemini-2.5-Flash: 68.5 (+10.2% over best baseline)
  • DocLens + Claude-4-Sonnet: 64.8 (+3.1% over best baseline)
  • Chart-specific gains over OCR-augmented baseline: +10.9% (Pro), +23.5% (Flash)
  • Ablation (Gemini-2.5-Pro, MMLongBench-Doc): removing Lens Module drops 4.1%; removing Reasoning Module drops primarily on UNA queries
  • Page Navigator recall: 97.3% (Pro), 95.2% (Flash), 90.2% (Flash-Lite); vs. 71.1% for MDocAgent and 89.0% for SimpleDoc

Limitations

  • Relies on proprietary models (Gemini-2.5-Pro/Flash, Claude-4-Sonnet); no open-source backbone evaluation reported.
  • High inference cost for Page Navigator, which must process all \(N\) pages (avg. 49.4 on MMLongBench-Doc); mitigated but not eliminated by hybrid backbones.
  • OCR and layout detection tools are external dependencies; their failure modes (e.g., scanned vs. born-digital PDFs, non-standard layouts) are not analyzed.
  • Evaluation limited to two English-language benchmarks; cross-lingual or domain-shift generalization is untested.
  • The paper text is truncated; the discussion of test-time scaling (Appendix D.1) and full implementation details are not available for review.

Relevance to Harnesses / Meta-Harnesses

DocLens is a concrete instance of a multi-agent harness architecture: it orchestrates four specialized sub-agents (Page Navigator, Element Localizer, Answer Sampler, Adjudicator) with explicit tool integrations (OCR, layout detection, cropping) and a structured control flow (retrieve → localize → sample → adjudicate). The two-module decomposition (\(f_{\text{extract}} \circ f_{\text{generate}}\)) exemplifies the harness pattern of separating concerns across agents with defined input/output contracts. The sampling-adjudication loop within the Reasoning Module is a micro-harness pattern—parallel generation followed by a judge agent—directly analogous to sample-and-verify patterns in broader agentic frameworks. The hybrid-backbone design, where cheaper models handle high-volume retrieval and expensive models handle low-volume reasoning, demonstrates cost-aware orchestration that is a key engineering concern in production meta-harnesses.