DAVE: A VLM Vision Encoder for Document Understanding and Web Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Document-aware vision encoder boosts VLM structural understanding for web agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DAVE is a vision encoder purpose-built for VLMs used in document understanding and web agent tasks, addressing the structural and spatial blindness of standard contrastive encoders (e.g., SigLIP). Its two-stage training pipeline—MAE-based self-supervised pretraining on 20M unlabeled images followed by supervised autoregressive pretraining—combined with weight-space merging and ensemble fusion with SigLIP2, yields a decoder-agnostic encoder that improves over SigLIP 2 by 10.5% on document/web VL benchmarks and 5% on the Mind2Web web agent benchmark.

Problem

Predominant VLM vision encoders (CLIP/SigLIP-style) learn high-level contrastive features that lack the low-level structural and spatial precision required to parse document layouts, OCR text, and navigate web UIs. DINO-style encoders have richer low-level features but are tuned to natural images and transfer poorly to documents and UIs. Additionally, training specialist encoders introduces two compounding problems: (i) annotation scarcity for document/web images (OCR-based pipelines bottleneck scalability and quality), and (ii) encoder-decoder coupling, where a specialist encoder pretrained with one text decoder degrades severely when paired with a different decoder.

Method

Stage 1 — Modified MAE SSL. A ViT-L16-384 encoder is trained with MAE on 20M unlabeled images (10M PDF pages from DocFM, 10M web screenshots from Common-Web). Standard MAE uses per-patch normalized reconstruction; document/web images have characteristically low inter-patch variance, which destabilizes this objective. DAVE instead reconstructs raw pixel values: $\(\mathcal{L}_{\text{MAE-pixel}} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \|f_\theta(\tilde{x})_i - x_i\|_2^2\)$ This stabilizes training at scale without additional hyperparameter tuning.

Stage 2 — Supervised Autoregressive Pretraining. Using ~2M labeled samples (OCR, layout extraction, chart-to-markdown, web UI grounding), the encoder is trained within a VLM (encoder + MLP projector + LLM) on autoregressive text generation.

• Multi-decoder training + weight-space merging. To avoid coupling to a single decoder, \(n\) encoder instances \(\{\phi_1, \ldots, \phi_n\}\) are trained independently paired with QWen2.5-0.5B, Phi-4-mini-Instruct, and Granite-3.1-3B. A set of learned scalar coefficients \(\{\alpha_i^{(j)}\}\) combines corresponding weights via weighted sum \(\theta_{\text{merge}}^{(j)} = \sum_{i=1}^n \alpha_i^{(j)} \theta_i^{(j)}\), \(\alpha_i \in [0,1]\). The coefficients are optimized by minimizing MSE distillation loss to all \(n\) teacher encoders while keeping encoder parameters frozen.
• Ensemble training. A frozen generalist encoder (SigLIP2) is concatenated with the specialist: \(\phi_{\text{DAVE}}(x) = \text{Concat}(\phi_{\text{gen}}(x), \phi_{\text{spec}}(x))\). This encourages \(\phi_{\text{spec}}\) to focus on structural/spatial features while SigLIP2 retains high-level semantics.

Key Contributions

• DAVE: a specialist vision encoder for document/web VLM tasks trained entirely without large-scale manual annotation.
• Modified MAE objective (raw pixel reconstruction) that stabilizes SSL on low-inter-patch-variance document and web images.
• Distillation-based weight-space merging to produce a decoder-agnostic encoder from multiple decoder-specific pretrained instances.
• Ensemble training that fuses structural specialist features with frozen generalist (SigLIP2) features at pretraining time.
• Demonstration that structural/spatial visual features matter more than broad semantic features for web navigation tasks.

Results

• Document/Web VL benchmarks (Llama-3.2-3B backbone, 8 benchmarks): DAVE outperforms SigLIP 2 by 10.5% average; e.g., DocVQA 82.1 vs. 72.1, OCRBench 62.2 vs. 51.5, InfoVQA 50.2 vs. 40.6, Screenspot-V2 63.1 vs. 51.8.
• Mind2Web web agent (Llama-3.2-3B): DAVE achieves Cross-Task Element Acc 30.8 / Step SR 26.1 vs. Dolphin (best baseline) 24.0 / 19.6 — approximately +5% over the strongest baseline.
• Classic document tasks: DocLayNet mAP 74.1 vs. SigLIP 2 70.8; DocBank mAP 56.9 vs. SigLIP 2 51.7; RICO-SCA 92.8 vs. SigLIP 2 93.3 (slightly below).
• General VQA (MMMU, RealWorldQA): no meaningful degradation relative to SigLIP 2, confirming ensemble preserves general semantics.
• Ablation (Stage 2, Doc avg): Scratch decoder 52.2 → pretrained decoder 55.1 → + ensemble 55.6 → + merge 63.4; weight merging gives the largest single jump.
• Merge strategy ablation: No merge 55.6 / 53.0 → average merge 62.8 / 67.7 → Fisher merge 60.3 / 67.0 → learned coefficients 63.4 / 68.2 on Doc/Web.

Limitations

• DAVE's embedding dimension is 2× SigLIP2 (concatenated features), making downstream pooling with simple MLP heads harder; this likely causes the slight underperformance on RICO-SCA (semantics-heavy classification).
• Supervised stage data quality is still partially bounded by OCR model accuracy for the self-processed arXiv samples.
• Web agent evaluation is limited to Mind2Web offline evaluation; no online/live browser interaction is tested.
• The weight-merging distillation step requires an additional 20 epochs on unlabeled data, adding training cost.
• Generalization beyond English documents is not evaluated (SSL web data has no language filter, but supervised data is English-only).

Relevance to Agentic AI / LLM Agents

Web agents that operate via screenshots rely entirely on the VLM's ability to visually parse UI elements—buttons, input fields, menus—at precise spatial locations; DAVE's 5% gain on Mind2Web demonstrates that encoder quality is a meaningful lever for agent action accuracy, not just comprehension. The finding that structural/spatial features dominate semantic features for web navigation (self-supervised MAE encoders compete with SigLIP2) is a design insight directly applicable to building multimodal agent backbones. The weight-space merging technique addresses a practical deployment problem—specialist encoders can now be dropped into diverse agent VLM stacks without decoder-specific re-tuning. This work establishes a reusable encoder component paradigm for the agentic AI stack, analogous to how domain-adapted language models (e.g., code LLMs) serve specialized agent tasks.