Skip to content

Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering

🕒 Published (v1): 2026-07-08 09:13 UTC · Source: Arxiv · link

Why this paper was selected

Comparative domain-adapted VLMs for DocVQA — honest baselines across architectures for document understanding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper benchmarks 8 open-source VLMs (Qwen3-VL 2B/4B/8B, Qwen3.5 0.8B/2B/4B/9B, Gemma 3 4B) on Document VQA across three document domains—industrial documents (SP-DocVQA), infographics (InfographicsVQA), and slides (SlideVQA)—under zero-shot, fully supervised, and few-shot conditions. The central finding is that visual complexity, not lack of linguistic knowledge, is the primary bottleneck for cross-domain DocVQA. With only 50 target-domain samples, source-domain-finetuned models often match or exceed fully supervised baselines.

Problem

While VLMs achieve strong DocVQA performance on structured document layouts, their robustness to visually complex domains (infographics, slides) is poorly characterized. There is no systematic study comparing zero-shot generalization, intra-/inter-domain supervised finetuning, and few-shot transfer across document domains of varying layout complexity.

Method

Eight open-source VLMs from three families (Qwen3-VL, Qwen3.5 with MoE, Gemma 3) spanning 0.8B–9B parameters are evaluated on SP-DocVQA (50K QA pairs, structured industrial docs), InfographicsVQA (~30K QA pairs, complex visual layouts), and SlideVQA (single-hop subset, unstructured visual slides). All images are padded to \(1449 \times 1449\) px for uniform input. Supervised finetuning is done with LoRA (rank 32, \(\alpha=4\), dropout 0.2) for 3 epochs via the Unsloth library. Few-shot experiments continue finetuning from source-domain checkpoints for 3 more epochs on 5, 20, or 50 SlideVQA samples. Performance is measured by Average Normalized Levenshtein Similarity (ANLS).

Key Contributions

  • Comprehensive zero-shot evaluation of 8 VLMs across three structurally distinct DocVQA domains, revealing a large ANLS gap between structured (SP-DocVQA: up to 92.67) and visually complex domains (SlideVQA: up to 51.38).
  • Systematic intra- and inter-domain finetuning study showing that smaller models gain larger relative ANLS improvements from finetuning, while large models show diminishing returns.
  • Few-shot cross-domain transfer experiments demonstrating that 50 target-domain samples suffice to nearly match—or exceed—fully supervised intra-domain finetuning on SlideVQA.
  • Evidence that visual layout understanding, not factual/linguistic knowledge, is the primary limiting factor in DocVQA generalization.

Results

  • Zero-shot: Best models (Qwen3-VL 8B, Qwen3.5 9B) reach 91.98/92.67 ANLS on SP-DocVQA, but only 57.55/60.06 on InfographicsVQA and 49.63/51.38 on SlideVQA.
  • Supervised finetuning (intra-domain): Qwen3.5 9B finetuned on SP-DocVQA reaches 96.55 ANLS (+3.88 vs. zero-shot); Qwen3.5 2B gains +25.03 on InfographicsVQA after finetuning on that domain; Qwen3.5 0.8B gains +18.04 on SlideVQA after SlideVQA finetuning.
  • Smaller model scaling: Qwen3.5 0.8B improves +17.45 ANLS on SP-DocVQA after finetuning, the largest absolute gain across all settings.
  • Cross-domain finetuning: Models finetuned on InfographicsVQA generalize better to SlideVQA than those finetuned on SP-DocVQA, suggesting visual layout similarity between the two complex domains.
  • Few-shot transfer (50 samples): Several models finetuned on a source domain then adapted with 50 SlideVQA samples match or exceed fully supervised SlideVQA baselines; Qwen3.5 9B is an exception, degrading with 5 samples before recovering.
  • Gemma 3 4B anomaly: After finetuning on SP-DocVQA, InfographicsVQA ANLS decreases by 2.24; after SlideVQA finetuning, SP-DocVQA ANLS decreases by 0.30, indicating catastrophic forgetting.

Limitations

  • Hardware constraints restricted model selection to ≤9B parameters; no large-scale models (e.g., 72B+) were evaluated.
  • SlideVQA multi-hop questions were excluded, making the slide evaluation easier than the full benchmark.
  • Only LoRA finetuning was tested; full fine-tuning or other PEFT methods were not compared.
  • Few-shot experiments used SlideVQA as the sole target domain; generalizability of the 50-sample finding to other target domains is untested.
  • Gemma 3's tendency to hallucinate under domain shift is noted but not mechanistically analyzed.
  • The paper text is truncated before the few-shot conclusions are fully stated.

Relevance to Vision-Language Models

This study directly measures the cross-domain generalization ceiling of current open-source VLMs in a challenging multimodal task, providing a diagnostic that visual encoder capacity—not language model depth—is the bottleneck for complex layout understanding. The finding that 50 target-domain samples largely close the gap between cross-domain and fully supervised performance has practical implications for efficient domain adaptation of VLMs. The systematic LoRA finetuning comparison across model families and scales offers a concrete recipe for deploying VLMs in real-world document processing pipelines. Results confirm that parameter scale follows a law of diminishing returns under supervised adaptation, making small-to-mid-scale VLMs competitively viable for specialized DocVQA deployment.