Skip to content

ColPali: Efficient Document Retrieval with Vision Language Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ColPali repurposes a Vision-Language Model (PaliGemma-3B) as a multi-vector document retrieval engine by treating each image patch token as an individual embedding and applying ColBERT-style late interaction scoring between query tokens and patch embeddings. This eliminates the fragile OCR/layout-detection/chunking pipeline required by text-centric retrieval systems. Paired with the new ViDoRe benchmark, ColPali achieves state-of-the-art page-level retrieval on visually rich documents while indexing pages ~18× faster than captioning-augmented pipelines.

Problem

Modern document retrieval pipelines convert PDFs to text via OCR, layout detection, and chunking before embedding—a process that is slow, brittle, and discards critical visual cues (figures, tables, infographics, fonts, layouts). Existing contrastive VLMs produce only single-vector embeddings and underperform text pipelines on document retrieval. No benchmark existed to evaluate retrieval systems on visually rich documents end-to-end across domains and languages.

Method

ColPali fine-tunes PaliGemma-3B (SigLIP vision encoder + Gemma-2B LLM) using LoRA adapters to produce ColBERT-style multi-vector embeddings directly from page images. Each of the 1,024 image patch tokens output by the LLM backbone is projected to a shared 128-dimensional embedding space (same as text token embeddings). At query time, a late interaction (MaxSim) score is computed between each query token embedding and all patch embeddings: LI(q,d) = Σᵢ maxⱼ ⟨Eq⁽ⁱ⁾ | Ed⁽ʲ⁾⟩. Training uses an in-batch contrastive loss with pairwise cross-entropy on hard negatives over 118,695 query-page pairs (63% public academic datasets + 37% VLM-synthesized pseudo-questions from web-crawled PDFs). Document pages are embedded in a single forward pass from their raw image with no preprocessing. Token pooling (hierarchical mean, pool factor 3) can reduce stored vectors by 66.7% with only 2.2% average performance loss.

Key Contributions

  • ViDoRe benchmark: page-level retrieval benchmark spanning 10 tasks across 5 domains, 2 languages (EN/FR), and multiple visual modalities (text, tables, figures, infographics), with both academic (repurposed VQA) and practical (LLM-generated queries over domain PDFs) splits.
  • ColPali model: end-to-end trainable VLM-based retriever producing per-patch multi-vector embeddings, eliminating OCR/layout preprocessing entirely.
  • Training dataset: 118,695 query-page pairs released openly.
  • Interpretability: late interaction heatmaps reveal which image patches are most salient per query token, providing spatial grounding for retrieval decisions.
  • Ablation framework: systematic comparison of BiSigLIP, BiPali, and ColPali variants isolating contributions of task-specific data, LLM contextualization, and multi-vector representation.

Results

All numbers are nDCG@5 on ViDoRe unless noted:

  • ColPali overall: 81.3 avg nDCG@5 vs. best text baseline Unstructured+OCR/BGE-M3 at 66.1 (+15.2 points).
  • vs. Unstructured+Captioning/BGE-M3: 81.3 vs. 67.0 (+14.3 points).
  • Visually complex tasks: InfoVQA 54.4 vs. 32.9 (+21.5), ArxivQA 79.1 vs. 35.7/40.1, TabFQuAD 83.9 vs. 71.9.
  • Contrastive VLMs (SigLIP vanilla): 51.4 avg; ColPali exceeds by ~30 points.
  • Indexing latency: ColPali 0.39 s/page vs. PDF parser pipeline 7.22 s/page (~18× faster); SigLIP 0.12 s/page.
  • Query latency: ColPali ~30 ms vs. BGE-M3 ~22 ms (both acceptable for industrial use).
  • Token pooling at factor 3: 66.7% vector reduction, 97.8% performance retained on average.
  • ColQwen2-VL (Qwen2-VL 2B backbone, same training): +5.3 nDCG@5 over ColPali, demonstrating generative VLM quality correlates with retrieval quality.
  • Out-of-domain (DocMatix-only training): −2.2 nDCG@5 vs. ColPali, still >12 points above closest baseline.
  • Storage: 257.5 KB per page at D=128.

Limitations

  • Multi-vector storage footprint is larger than single-vector bi-encoder approaches (257.5 KB/page), requiring compression/clustering for very large corpora.
  • Late interaction query latency (~30 ms) scales with corpus size, though optimized engines (PLAID) mitigate this.
  • Training dataset is English-only by design; multilingual generalization relies on Gemma-2B's pretraining, resulting in some performance gaps on French tasks without targeted fine-tuning.
  • Unfreezing the vision encoder during fine-tuning slightly degrades performance, suggesting the contrastive training signal at this scale is insufficient to update the full vision backbone beneficially.
  • Reducing image patches (512 vs. 1024) causes −24.8 nDCG@5, indicating performance is sensitive to resolution; memory/performance tradeoff is non-trivial.
  • Evaluation corpus sizes are capped (max 1,000 pages per task) to accommodate slow baseline pipelines, limiting conclusions about very large-scale retrieval.

Relevance to Vision-Language Models

ColPali is a direct demonstration that generative VLMs trained for captioning and visual QA can be repurposed into high-quality dense retrievers by tapping their patch-level token representations, bypassing the need for text extraction entirely. The key insight—that a VLM's output patch embeddings already live in a space aligned with text embeddings via multimodal pretraining—enables applying the ColBERT late interaction framework cross-modally. This opens a new "retrieval in vision space" paradigm relevant to RAG systems, showing that VLM architectural choices (patch count, LLM size, full-block attention on prefix) directly determine retrieval quality. The result also validates that improvements in generative VLM benchmarks transfer to retrieval, making future VLM development doubly impactful for downstream document understanding pipelines.