Skip to content

RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; RAVENEA: retrieval-augmented visual culture understanding benchmark

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RAVENEA is a benchmark for evaluating retrieval-augmented generation (RAG) applied to visual culture understanding in VLMs, covering two tasks—culture-centric VQA (cVQA) and culture-informed image captioning (cIC)—across eight countries with 1,868 instances paired to 11,396 human-ranked Wikipedia documents. Evaluation of 7 multimodal retrievers and 17 VLMs shows culture-aware retrieval yields average gains of +6% on cVQA and +11% on cIC, with disproportionate benefit to lightweight models (≤8B parameters).

Problem

Prior work on cultural understanding via RAG operates in text-only settings. Existing multimodal culture datasets probe VLMs' memorized knowledge rather than their ability to consume externally retrieved cultural context. It is therefore unknown whether current multimodal retrievers can reliably surface culturally relevant documents, and whether RAG can meaningfully improve VLM cultural comprehension in multimodal scenarios.

Method

Benchmark construction: Images from CVQA (culture-related VQA) and CCUB (culturally contextualized captions) are paired with Wikipedia documents via a two-stage pipeline: (1) GPT-4o generates culturally grounded captions per image; (2) BM25 retrieves the top-10 candidate documents from a 6M-article Wikipedia corpus. Human annotators then re-rank documents along three binary dimensions—country association, topic alignment, and explicit visual representation—yielding 18,680 human-labeled image-document pairs. Final dataset: 1,868 instances (2,331 cVQA image-text questions, 655 cIC image-caption pairs), eight countries, eleven categories.

Culture-Aware Contrastive (CAC) learning: Fine-tuning CLIP/SigLIP retrievers with a composite loss: $\(\mathcal{L}_\text{CAC} = \frac{1}{3}(\mathcal{L}_\text{CultureClassify} + \mathcal{L}_\text{Rank} + \mathcal{L}_\text{Diversity})\)$ - \(\mathcal{L}_\text{CultureClassify}\): binary cross-entropy on per-document relevance labels via sigmoid over cosine similarity. - \(\mathcal{L}_\text{Rank}\): margin ranking loss promoting relevant documents over irrelevant ones. - \(\mathcal{L}_\text{Diversity}\): entropy-based penalty discouraging redundant positive text embeddings.

RegionScore metric: Binary indicator checking whether a generated caption contains the country name or a demonym, averaged over instances: \(\text{RegionScore}_I = \frac{1}{|I|}\sum_{I_i \in I} R(g^{(i)}, I_i)\). This metric achieves Kendall's \(\tau = 0.442\) correlation with human judgments vs. \(-0.316\) for CIDEr and \(-0.172\) for ROUGE-L.

Key Contributions

  • RAVENEA benchmark: 1,868 instances, 8 countries, 11 categories, 18,680 human-labeled image-document pairs, supporting cVQA and cIC tasks.
  • CAC learning: a three-term contrastive loss for fine-tuning CLIP/SigLIP-style retrievers with explicit cultural supervision.
  • RegionScore: a novel, human-validated metric for geographic specificity in culture-aware captioning (\(\tau = 0.442\), only metric with significant correlation).
  • Systematic evaluation: 7 retrievers Ă— 17 VLMs (2B–78B, open-source and proprietary) revealing scaling behavior and cross-cultural performance variation.

Results

  • Retrieval: Ravenea-CLIP (fine-tuned CLIP-L/14@224px) achieves P@1 = 72.05% vs. 60.87% frozen baseline; MRR = 82.17% vs. 75.44%; nDCG@5 = 84.09% vs. 78.09%—new SOTA across all metrics.
  • cVQA: Ravenea-CLIP improves average accuracy from 67.1% (no RAG) to 73.3% (+6.2% absolute), averaged over all 17 VLMs.
  • cIC: Ravenea-CLIP improves average RegionScore from 57.5% to 68.8% (+11.3% absolute).
  • Scaling: Lightweight models benefit most; Qwen3-VL-2B gains +31.6% on cVQA with Ravenea-CLIP; larger models (27B–78B) see only modest gains.
  • Without RAG: InternVL3-78B reaches 84.2% cVQA accuracy; Gemma3-27B reaches 75.5% cIC RegionScore—strong ceilings that RAG still improves for smaller models.
  • Metric validity: RegionScore is the only evaluated metric significantly correlated with human judgment (\(\tau = 0.442\), \(p < 0.05\)); standard metrics (ROUGE-L, CIDEr, BERTScore, CLIPScore) show no significant correlation or negative correlation.

Limitations

  • Covers only eight countries (sampled from 30); performance disparities across countries suggest uneven cultural representation that the benchmark partially but incompletely addresses.
  • RegionScore is a binary, surface-form metric—it captures geographic mention but not deeper cultural accuracy or nuance in captions.
  • Human annotation is conducted only over the top-10 BM25 results; culturally relevant documents outside this candidate pool are not systematically captured (though annotators can suggest missing articles).
  • Culture-aware gains vary widely across countries and model families, making it difficult to generalize findings beyond the evaluated configurations.
  • Large proprietary and frontier VLMs see limited gains from retrieval, suggesting retrieval benefits may diminish as parametric cultural knowledge scales.

Relevance to Vision-Language Models

RAVENEA directly probes a known blind spot of VLMs—cultural understanding outside dominant (Western) training distributions—by introducing a controlled RAG evaluation setup with human-quality retrieval supervision. The finding that culture-aware fine-tuning of contrastive retrievers (CAC loss) consistently unlocks downstream gains in both VQA and captioning is directly actionable for practitioners deploying VLMs in culturally diverse settings. The RegionScore metric addresses a measurement gap that renders standard captioning metrics uninformative for cultural evaluation. Critically, the scaling analysis reveals that retrieval augmentation is most impactful for small VLMs, suggesting a practical path to culturally capable lightweight models without the cost of scaling parameters.