Skip to content

ARIAL: An Agentic Framework for Document VQA with Precise Answer Localization

🕒 Published (v1): 2025-11-22 21:09 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ARIAL is a modular agentic framework for Document VQA that decomposes queries into tool calls across OCR, retrieval, generative QA, and spatial grounding modules, all orchestrated by an LLM-based planning agent. It achieves state-of-the-art results on DocVQA (88.7 ANLS, 50.1 mAP@IoU), FUNSD, CORD, and SROIE, simultaneously improving textual accuracy and precise answer bounding-box localization over prior monolithic approaches.

Problem

Existing Document VQA systems either achieve strong textual accuracy without reliable spatial grounding, or sacrifice answer quality for interpretability. Monolithic multimodal transformers (e.g., DLaVA) treat localization as an end-to-end prediction problem within a single large model, producing opaque reasoning and limiting error diagnosis. Standard metrics like ANLS ignore spatial correctness entirely.

Method

ARIAL employs a central Planner Agent (LLaMA 4 Scout, fine-tuned via behavioral cloning on 50 demonstration traces) that dynamically routes queries through four specialized modules following a sense-think-act paradigm:

  1. OCR — DB detector (ResNet-50) + TrOCR produces text segments \(\{(T_i, B_i)\}_{i=1}^N\) with bounding boxes.
  2. Retrieval — FindText(keywords) encodes segments with MiniLM-v6 (384-dim), retrieves top-\(k\) by cosine similarity + keyword match, reducing QA context to relevant spans.
  3. Generative QA — AskQA(context, Q) calls Gemma 3-27B (fine-tuned on 70k document QA pairs) to generate answers from retrieved context.
  4. Grounding — GroundAnswer(A) aligns generated answer \(A\) back to OCR bounding boxes via exact match, fuzzy match (Levenshtein \(\leq 2\)), or semantic similarity (\(\geq 0.85\) cosine), merging multi-span boxes into unified region \(B_A\).

The agent constructs action sequences \(\{a_1, \ldots, a_n\}\) adaptively, terminating on confident grounded output.

Key Contributions

  • Agentic Document VQA architecture that decomposes queries into tool calls, enabling tool reuse, error tracing, and transparent reasoning chains.
  • Explicit, precise answer localization producing bounding boxes anchored to OCR-detected spans, evaluated with mAP@IoU 0.50:0.95.
  • Retrieval-augmented generation over OCR segments to focus QA on relevant context and suppress hallucination.
  • SoTA results on four benchmarks under both ANLS and spatial mAP metrics.

Results

  • DocVQA: 88.7 ANLS (+2.8 vs. DLaVA), 50.1 mAP@IoU (+3.9 vs. DLaVA OCR-Free; +15.2 vs. DLaVA OCR-Dependent)
  • FUNSD: 90.0 ANLS (+2.4 vs. DLaVA), 50.3 mAP@IoU (+4.8 vs. DLaVA OCR-Free; +18.3 vs. DLaVA OCR-Dependent)
  • CORD: 85.5 ANLS (+1.1 vs. DLaVA), 60.2 mAP@IoU (+2.3 vs. DLaVA OCR-Free; +12.2 vs. DLaVA OCR-Dependent)
  • SROIE: 93.1 ANLS (+1.7 vs. DLaVA)
  • Ablation: removing LLM planner (heuristic fixed pipeline) costs −5.1 ANLS and −5.9 mAP on DocVQA; removing retrieval costs −2.5 ANLS and −1.6 mAP
  • Average inference latency: 3.2 s/query vs. 1.2 s for DLaVA and 0.4 s for DocLayLLM

Limitations

  • Hard OCR dependency: severe noise, degradation, or non-standard fonts degrade the entire pipeline.
  • Sequential tool execution bottlenecks throughput (3.2 s/query); parallelization or caching not yet implemented.
  • Planner fine-tuned on only 50 demonstration traces — generalization to novel document types or query patterns may be limited.
  • No multi-document or cross-document reasoning support.
  • Model distillation not explored; deployment on resource-constrained hardware is impractical.

Relevance to Harnesses / Meta-Harnesses

ARIAL is a concrete instantiation of the harness pattern: a meta-controller (LLM planner) dynamically selects and sequences specialized tool-agents (OCR, retrieval, QA, grounding) rather than embedding all logic in a single monolithic model. The ablation study directly validates the harness hypothesis — replacing the adaptive LLM planner with a fixed-order heuristic pipeline causes the largest performance drop of any component, quantifying the value of dynamic orchestration over static chaining. The modular design also demonstrates a key harness property: independent component upgradability without retraining the orchestrator. For researchers building meta-harnesses, ARIAL is a strong case study in how behavioral cloning on a small set of demonstration traces (~50) can suffice to train a competent planner agent for a well-scoped tool suite.