Skip to content

A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends

🕒 Published (v1): 2025-07-14 02:10 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This survey systematically reviews MLLM-based Visually Rich Document Understanding (VRDU), covering OCR-dependent and OCR-free framework architectures, multimodal representation strategies across text/visual/layout modalities, training paradigms, and emerging trends including Retrieval-Augmented Generation and agentic frameworks. It taxonomizes ~40 models (2022–2026) and identifies core challenges blocking scalable deployment. The work serves as a structured roadmap for advancing MLLM-based VRDU toward multi-page, multilingual, and agentic settings.

Problem

Prior surveys either broadly covered MLLM capabilities without VRDU specialization, or addressed isolated VRDU subtasks (layout analysis, QA, relation extraction) without a unified MLLM-centric treatment. General-domain MLLMs applied naively to VRDs suffer from weak layout inductive bias, sensitivity to OCR noise, and hallucination on knowledge-intensive document tasks. A systematic analysis of MLLM-specific architectural choices—representation, training, and inference—was absent.

Method

The survey organizes MLLM-based VRDU along four orthogonal axes:

  1. Framework Architecture: OCR-dependent (external parsers feed text+bounding boxes into multimodal encoders + LLM) vs. OCR-free (end-to-end vision encoder → compressor → LLM).
  2. Multimodal Representation: Three modalities—text (LLM-injected, auxiliary encoder, training objective), visual (low/high-resolution encoding, visual token compression via Q-Former, Resampler, or convolutional layout-aware modules), and layout (positional encoding, layout-as-prompt verbalization, training-time grounding tasks)—combined via neural-based, target-oriented, or prompt-based fusion.
  3. Training Paradigms: Self-supervised pretraining (masked modeling, cross-modality alignment, text spotting), instruction tuning on large-scale QA/KIE datasets (synthetic and real), and supervised fine-tuning (SFT) for domain adaptation; LLM backbones are selectively frozen or unfrozen per stage.
  4. Inference Prompt Setting: ICL+layout, CoT, bounding-box-augmented queries, task-conditioned prompts.

Key Contributions

  • Unified taxonomy of MLLM-based VRDU covering OCR-dependent and OCR-free architectures, spanning ~40 models from 2022–2026.
  • Detailed comparative table (Table 1) encoding modality inputs, LLM/vision encoder choices, pretraining/instruction-tuning/fine-tuning stages, single/multi-page support, and prompt formats per model.
  • Systematic breakdown of multimodal representation strategies across text, visual, and layout modalities, including compression mechanisms for long visual sequences.
  • Analysis of training strategy interactions (frozen vs. trainable LLM, adapter modules, LoRA, linear projectors).
  • Discussion of open challenges: data scarcity in low-resource/scanned domains, multi-page coherence, multilingual support, cumulative OCR error propagation.
  • Forward-looking coverage of RAG integration and agentic document-understanding frameworks (e.g., MDocAgent) as emerging trends.

Results

This is a survey; no new benchmark numbers are reported by the authors. The paper notes: - General-domain MLLMs show "promising performance" on VRDU tasks but exhibit fundamental limitations (layout bias, OCR sensitivity, hallucination) documented across cited works. - Instruction tuning on large-scale datasets "substantially enhances zero-shot performance" per reviewed literature, but at high resource cost. - Synthetic QA datasets from OCR+LLM pipelines yield low-quality pairs in low-resource domains, degrading zero-shot results. - Quantitative comparisons across individual cited models are deferred to Appendix C (not included in the provided excerpt).

Limitations

  • Scope restricted to document-level MLLM-based understanding; excludes multi-document retrieval/reasoning, non-LLM methods, and general MLLMs without VRD-specific adaptation.
  • Paper text is truncated; sections on RAG, agentic frameworks (Section 7), inference details (Section 6), and quantitative appendices (Appendix C) are not present in the excerpt, limiting full assessment.
  • Survey reflects a snapshot; the field is evolving rapidly (multiple 2025/2026 preprints cited).
  • No empirical validation of the proposed taxonomy or coverage claims.

Relevance to Harnesses / Meta-Harnesses

The survey explicitly identifies agentic frameworks as an emerging trend in VRDU, with MDocAgent (a multi-agent pipeline for multi-page document QA) listed as a representative system—directly relevant to harness-oriented work. The coverage of pipeline composition (OCR tools → encoders → compressors → LLMs → answer decoders) maps onto the modular, composable structure typical of meta-harnesses. The discussion of RAG integration (retriever + reader orchestration over document corpora) and multi-page coherence handling describes sub-harness coordination patterns for document-scale tasks. For researchers building document-understanding harnesses, this survey provides a structured inventory of interchangeable components (vision encoders, adapters, fusion modules, training stages) and flags where pipeline brittleness arises (OCR error propagation, token compression fidelity, cross-page context loss).