Skip to content

AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AlignVLM introduces ALIGN, a novel VLM connector that maps visual features to a weighted average (convex combination) of the LLM's pretrained text embeddings rather than projecting them into free latent space. This constrains visual representations to lie within the convex hull of the LLM's text embedding space, reducing cross-modal misalignment. The approach is especially effective for document understanding and low-resource regimes, outperforming MLP, Perceiver Resampler, and Ovis connectors under identical training conditions.

Problem

Existing shallow-fusion VLM connectors (MLPs, Perceiver Resampler, Ovis) lack inductive bias to keep projected visual features within the distribution of the LLM's pretrained text embedding space. Projections learned from scratch can produce out-of-distribution visual tokens that cause cross-modal misalignment, require large datasets to compensate, and are brittle to noise. Deep-fusion methods mitigate this but at the cost of significantly increased parameter counts and computational overhead.

Method

ALIGN replaces the standard connector with a two-step module: 1. A linear projection W₁ ∈ ℝ^{D×d} maps visual patch features F to the LLM's token embedding dimension D. 2. A second linear layer W₂ ∈ ℝ^{V×D} (initialized from the LLM's language-model head) followed by LayerNorm and softmax produces a probability simplex P_vocab over the full vocabulary V. 3. The final visual representation is F′_align = P_vocab^⊤ · E_text — a convex combination of the LLM's frozen text embedding matrix E_text.

Because each output vector is a convex combination of existing text embeddings, it is mathematically constrained inside the convex hull of the LLM's text latent space. W₂ is initialized from the LM head, seeding the connector with linguistic priors rather than training from random weights. Training proceeds in three stages: (1) image-text pretraining on 8.1M CC-12M pairs (full model unfrozen); (2) document understanding fine-tuning on BigDocs-7.5M (full model); (3) instruction tuning on DocDownstream (vision encoder frozen).

Key Contributions

  • ALIGN connector: maps visual features to a weighted average of LLM text embeddings, enforcing convex-hull membership in the LLM's text latent space.
  • AlignVLM family (Llama-3.2-1B/3B, Llama-3.1-8B backbones) achieving SOTA among base VLMs trained on the same data.
  • Empirical demonstration that ALIGN is significantly more data-efficient and noise-robust than MLP and other shallow-fusion connectors.
  • Analysis showing ALIGN relies on only ~3.4K of 128K vocabulary embeddings, suggesting a viable pruning path for further efficiency.

Results

  • Connector ablation (high-resource, Llama-3.2-3B): ALIGN 58.81% avg. vs. MLP 53.06%, Perceiver Resampler 50.68%, Ovis 54.72% across 9 document benchmarks.
  • Low-resource ablation (LLaVA-NeXT 779K): ALIGN 56.88% avg. vs. MLP 53.59%, Perceiver 53.17%, Ovis 54.18%, H-Reducer 54.33%, HoneyBee 54.79%; larger margins on document tasks than general VL tasks.
  • AlignVLM-3B vs. base VLMs (same training data): outperforms Qwen2-VL-2B by up to 9.22%; surpasses DocOwl1.5-8B (4B more parameters) at 58.81% vs. ~57.56%.
  • AlignVLM-8B vs. base VLMs: 2.62% improvement over Llama3.2-11B despite fewer parameters; outperforms InternVL-2.5-8B; competitive with Qwen2.5-VL-7B.
  • Noise robustness (σ=3 Gaussian on visual features): ALIGN drops only 1.67% avg.; MLP drops 25.54%. Cosine distance from original features: ALIGN 0.0036 vs. MLP 0.3938.
  • Embedding sparsity: using only 3.4K of 128K vocabulary embeddings yields negligible performance degradation (e.g., DocVQA 79.40 vs. 79.63).

Limitations

  • Evaluated primarily on document understanding; generalization to open-domain visual tasks (e.g., natural scenes, VQA) is shown to be competitive but narrower gains were demonstrated.
  • Maximum 9 image tiles limits resolution; high-resolution or very long documents may require further engineering.
  • Training requires 64 H100 GPUs (8 nodes), limiting reproducibility for smaller research groups.
  • Future work explicitly acknowledges the need to train on more diverse instruction-tuning datasets to broaden domain coverage.
  • The convex combination over the full 128K vocabulary introduces a softmax over a large matrix; though pruning to 3.4K is shown viable, it was not incorporated into the main training pipeline.

Relevance to Vision-Language Models

AlignVLM directly addresses the fundamental connector design problem in the standard vision encoder → connector → LLM VLM architecture, which is central to essentially all current open-source VLMs (LLaVA, InternVL, Qwen-VL, etc.). The key insight — that visual features should be expressed as convex combinations of existing LLM text embeddings rather than unconstrained projections — offers a principled inductive bias grounded in the geometry of the LLM's pretrained embedding space. This is directly relevant to researchers tracking VLMs because it challenges the dominant MLP-connector paradigm with a parameter-efficient alternative that improves both performance and robustness without modifying the LLM architecture. The demonstrated gains in low-resource settings are particularly significant for academic and smaller-scale VLM research where data and compute are constrained.