DAVE: A VLM Vision Encoder for Document Understanding and Web Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

DAVE encoder captures low-level spatial structure improving document VLM performance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DAVE is a specialized vision encoder for VLMs that addresses the structural/spatial deficiency of standard contrastive encoders (e.g., SigLIP) on document and web images. It uses a two-stage pipeline—MAE-based self-supervised pretraining on 20M unlabeled document/web images, followed by supervised autoregressive pretraining with weight-space merging and generalist-encoder ensemble fusion—to produce a decoder-agnostic, domain-specialized encoder. DAVE outperforms SigLIP 2 by an average of 10.5% across eight document and web benchmarks and improves web agent performance (Mind2Web) by 5% over the best baseline.

Problem

Contrastive vision encoders dominant in VLMs (CLIP, SigLIP) lack the structural and spatial low-level features required for document parsing and web UI understanding. DINO-style SSL encoders provide low-level features but are optimized for natural images and transfer poorly to documents and UIs. Encoder-decoder document specialists (Pix2Struct, Dolphin) capture domain-specific structure but forfeit general visual semantics, and encoders trained with one LLM decoder overfit to that decoder, degrading performance when paired with others.

Method

Stage 1 — SSL on document/web images. A ViT-L/16-384 encoder is trained with a modified MAE objective that reconstructs raw pixel values instead of patch-normalized targets: $\(\mathcal{L}_{\text{MAE-pixel}} = \frac{1}{|M|}\sum_{i \in M} \|f_\theta(\tilde{x})_i - x_i\|_2^2\)$ The modification is motivated by document/web images having characteristically low inter-patch variance, which destabilizes the standard normalized MAE loss (Eq. 1). Training covers 20M images (10M PDF pages from DocFM, 10M web screenshots from Common-Web) for 120K steps.

Stage 2 — Supervised multi-task autoregressive pretraining (~2M samples). - Ensemble training: A frozen SigLIP2 (generalist) encoder \(\phi_\text{gen}\) is concatenated with the specialist encoder \(\phi_\text{spec}\): \(\phi_\text{DAVE}(x) = \text{Concat}(\phi_\text{gen}(x),\, \phi_\text{spec}(x))\). This forces \(\phi_\text{spec}\) to learn structural/spatial features while \(\phi_\text{gen}\) handles high-level semantics. - Multi-decoder training + weight-space merging: \(n\) encoder instances are each trained paired with a distinct LLM decoder (QWen2.5-0.5B, Phi-4-mini, Granite-3.1-3B). They are merged by learning scalar coefficients \(\{\alpha_i\}\) via a distillation loss: \(\mathcal{L}_\text{distill} = \frac{1}{n}\sum_{i=1}^n \|\hat{z}_i - z_i\|_2^2\), where \(\hat{z}_i\) are merged-encoder features and \(z_i\) are teacher-encoder features. Only the coefficients are optimized; encoder weights remain frozen.

Key Contributions

• DAVE: a purpose-built vision encoder for document understanding and web agents that integrates into standard VLM architectures.
• Modified MAE pixel-reconstruction loss that stabilizes self-supervised pretraining on low-variance document/web images.
• Multi-decoder training + learned-coefficient weight-space merging that yields a decoder-agnostic encoder.
• Ensemble pretraining paradigm that fuses specialist structural features with frozen SigLIP2 general semantics via early concatenation.

Results

• Document/web VQA (Llama-3.2-3B backbone): DAVE averages +10.5% over SigLIP 2 across 8 benchmarks (DocVQA: 82.1 vs. 72.1; OCRBench: 62.2 vs. 51.5; InfoVQA: 50.2 vs. 40.6; ChartQA: 63.1 vs. 51.8; Screenspot-V2: 36.6 vs. 36.9 ≈ tie).
• Document/web VQA (Qwen-2.5-7B backbone): Same trend holds; DocVQA 90.9 vs. 90.2, InfoVQA 60.2 vs. 55.2, ChartQA 82.5 vs. 75.9.
• Classic document tasks: DocBank mAP 56.9 vs. SigLIP2 51.7; DocLayNet mAP 74.1 vs. SigLIP2 70.8; RICO-SCA accuracy 92.8 vs. SigLIP2 93.3 (slight underperformance).
• Web agent — Mind2Web (Llama-3.2-3B): Cross-Task Step SR 26.1 vs. best baseline Dolphin 19.6; Cross-Task Element Acc 30.8 vs. 24.0; overall ~+5% over best baseline.
• Ablation (Stage 2 design): Adding pretrained decoder → ensemble → weight merging progressively improves average Doc accuracy from 52.2 → 55.1 → 55.6 → 63.4.
• Merging strategy: Learned coefficients (63.4 Doc, 68.2 Web) outperform average merging (62.8, 67.7) and Fisher merge (60.3, 67.0).

Limitations

• Slightly underperforms SigLIP2 on RICO-SCA screen classification (92.8 vs. 93.3), attributed to DAVE's doubled embedding dimension making MLP pooling harder.
• Multi-decoder training requires maintaining \(n\) separate encoder training runs and distillation, increasing pretraining cost.
• High-level semantics depend on a frozen SigLIP2 encoder; the specialist component alone is insufficient, meaning DAVE inherits SigLIP2's distribution biases.
• Evaluation of the full VLM (not just the encoder) uses only instruction-tuned baselines; the encoder's ceiling is bounded by decoder capacity and instruction tuning data.

Relevance to Vision-Language Models

DAVE directly addresses a structural gap in VLM design: the vision encoder, typically treated as a frozen pretrained backbone, is shown to be a primary bottleneck for document and agentic tasks. The model-merging approach is a practically important recipe for producing plug-in encoders compatible with heterogeneous VLM stacks without per-decoder finetuning. The ensemble paradigm of concatenating a specialist encoder with a frozen generalist encoder (SigLIP2) provides a principled way to add domain-specialized visual features to existing VLM pipelines at low marginal cost. For researchers tracking VLMs, DAVE establishes that targeted encoder pretraining on domain-specific unlabeled data—rather than scaling general image-text contrastive training—is a high-leverage intervention for structured visual understanding.