Skip to content

Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity

🕒 Published (v1): 2026-06-24 03:17 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV; benchmark exposing VLM retrieval failures under visual homogeneity at scale

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs near-human on single-document VQA degrade sharply when retrieval pools are visually homogeneous — existing benchmarks mask this by mixing document types. Invoice Haystack (1,500 invoices, 200 QA pairs, mean pairwise cosine similarity 0.73) exposes this gap, and the accompanying VL-RAG framework — fusing BGE-Large text embeddings with SigLIP/OpenCLIP visual embeddings plus a VLM reranker — recovers significant Recall@1 over vision-only and prior multimodal baselines.

Problem

Multi-document retrieval benchmarks (DocHaystack, InfoHaystack) aggregate visually diverse document types, inflating performance because negatives separate easily in embedding space. Real enterprise repositories consist of thousands of records sharing a small set of templates (e.g., invoices with 70–85% template reuse), causing embedding collapse in vision-only encoders — CLIP-family models map near-identical templates into indistinguishable representations despite differing semantic content. No benchmark previously stress-tested retrieval under this strong visual homogeneity at scale.

Method

Benchmark construction (4-stage pipeline): 1. Gemini 3 Pro anonymizes 2,000 raw invoices across 7 PII categories while preserving numerical consistency and layout geometry. 2. Gemini 2.5 Flash generates one discriminative QA pair per document (multi-field questions enforcing unique document identification). 3. GPT-5.2 filters to 200 high-quality, non-generic, visually-grounded pairs. 4. Expert human validators verify anonymization, question discriminativeness, and answer accuracy; corpus reduced to 1,500 images.

VL-RAG (hybrid dual-stream RAG): - Text stream: DeepSeek OCR → BGE-Large-En-v1.5 dense encoder → cosine similarity \(\text{Sim}_T\). - Vision stream: SigLIP (ViT-SO400M/14@384) and OpenCLIP (ConvNeXt-XXL@1024) encoders → cosine similarities \(\text{Sim}_S\), \(\text{Sim}_O\). - Score fusion: \(\text{Sim}_{\text{Avg}} = \frac{1}{3}(\text{Sim}_T + \text{Sim}_S + \text{Sim}_O)\); top-\(m\) candidates shortlisted. - VLM filter: Qwen3-VL-8B-Instruct performs binary yes/no verification per candidate; only confirmed documents are returned (\(m=10\), \(k \in \{1,3,5\}\)).

Key Contributions

  • Invoice Haystack benchmark: 1,500 anonymized invoice images × 200 discriminative QA pairs evaluated at corpus scales 500/1,000/1,500; mean pairwise SigLIP cosine similarity 0.73 vs. 0.31–0.38 for prior benchmarks.
  • Embedding collapse characterization: formal identification and naming of the failure mode where vision encoders map template-sharing documents to indistinguishable representations.
  • VL-RAG: training-free hybrid retrieval framework combining dense text, structural visual embeddings, and a VLM verification stage.
  • Ablation study validating the SigLIP + OpenCLIP ensemble and the \(m=10\) candidate pool size for the VLM filter.

Results

  • Invoice Haystack-500: VL-RAG 60.0% R@1 vs. V-RAG 46.5% (+13.5 pp); R@3 63.5% vs. 49.0%; R@5 63.5% vs. 49.0%.
  • Invoice Haystack-1000: VL-RAG 53.0% R@1 vs. V-RAG 43.0% (+10 pp).
  • Invoice Haystack-1500: VL-RAG 50.0% R@1 vs. V-RAG 40.0% (+10 pp).
  • DocHaystack-1000: VL-RAG 77.1% R@1 vs. V-RAG 75.2% (+1.9 pp).
  • InfoHaystack-1000: VL-RAG 84.5% R@1 vs. V-RAG 80.0% (+4.5 pp).
  • Vision-only CLIP on Invoice Haystack-500: 10.0% R@1; BM25 (text-only): 43.0% R@1 — both far below VL-RAG, confirming neither modality alone suffices.

Limitations

  • All 1,500 documents are invoices; generalization to other homogeneous template categories (receipts, standardized forms, legal contracts) is unverified.
  • VLM filter (Qwen3-VL-8B) adds non-trivial inference cost over pure embedding retrieval; latency at scale is not characterized.
  • OCR quality is a hard upstream dependency; degraded OCR on scanned or low-resolution invoices will directly harm the text stream.
  • Retrieval and QA evaluation is decoupled; end-to-end answer accuracy after retrieval is not the primary metric reported.
  • 200 QA pairs is a small evaluation set relative to corpus size; statistical confidence intervals are not reported.
  • Models are evaluated zero-shot; fine-tuned retrievers may change the competitive landscape significantly.

Relevance to Vision-Language Models

Invoice Haystack directly probes a failure mode of VLMs-as-retrievers: near-human single-document VQA accuracy does not transfer to multi-document settings under visual homogeneity, a regime that dominates enterprise document AI. The benchmark provides a controlled stress-test that is harder by design than DocHaystack/InfoHaystack (cosine similarity 0.73 vs. ≤0.38), making it useful for evaluating the next generation of document-specialized VLMs and multimodal embedding models. VL-RAG's VLM verification stage also illustrates a reranking paradigm — using a VLM as a binary judge over shortlisted candidates — that is directly applicable to any retrieval-augmented VLM pipeline. For researchers tracking VLMs in document AI, this work establishes a concrete empirical ceiling and a reproducible benchmark for measuring progress on homogeneous retrieval.