IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

South-Asian cultural VLM benchmark exposes Western-centric evaluation blindspots

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IndicVisionBench is the first large-scale benchmark for evaluating VLMs on Indian cultural and multilingual content, covering 5K images and 37K+ QA pairs across English and 10 Indic languages. It spans three tracks—VQA, OCR, and Multimodal Machine Translation—grounded in 13 India-specific cultural topics. Evaluation of 8 VLMs reveals substantial performance gaps, with proprietary models outperforming open-weight systems and all models struggling severely on adversarial and low-resource-language tasks.

Problem

Existing VLM evaluation benchmarks are Western-centric and fail to capture the linguistic and cultural diversity of regions like the Indian subcontinent. Prior work touching on India-specific contexts (CVQA, CulturalVQA, ALM-Bench) lacks unified coverage of both Indic cultural grounding and multilingual multimodal evaluation across medium-to-low resource languages. This leaves open questions about whether current VLMs generalize to culturally and linguistically diverse settings.

Method

The benchmark is constructed through three parallel pipelines:

• VQA: 615 crowd-sourced images + 3,502 expert-collected Creative Commons images across 13 cultural topics and all Indian States/UTs. Each image is annotated with human keywords, expanded into VLM-generated captions (Gemini-1.5-Flash/2.5-Flash), and yields 6 QA types per image: short-answer (×2), long-answer, MCQ, True/False, and adversarial (with embedded false assumptions). Translations into regional languages are human-corrected; a disjoint VQA-Parallel subset of 106 images is translated into all 10 Indic languages for cross-lingual analysis.
• MMT: The 106-image VQA-Parallel subset extended with human-verified English captions translated into 10 Indic languages with image context available.
• OCR: 876 document images from Wikisource Level-4 verified pages (printed and handwritten) across 10 Indic scripts, paired with gold transcriptions.

Evaluation uses Exact Match for MCQ/True-False, LLM-as-a-Judge (GPT-4o, 0–10 scale) for open-ended VQA, BLEU/RIBES for MMT, and ANLS (word- and character-level) + WER/CER for OCR.

Key Contributions

• First large-scale Indian-centric VLM benchmark: 5K images, 37K+ QA pairs, 13 cultural topics, 11 languages (English + 10 Indic).
• Three-track evaluation suite (VQA, OCR, MMT) with 6 QA types including adversarial questions probing cultural robustness.
• Paired parallel corpus across all 10 Indic languages enabling systematic cross-lingual variation analysis.
• Comprehensive evaluation of 8 VLMs (proprietary and open-weight, 7B to frontier scale) with statistical significance testing.
• Public release of benchmark, code, and annotations.

Results

VQA (English): - Gemini-2.5 leads all 6 question types (MCQ: 0.94, True/False: 0.95, Long-answer: 9.30/10, Adversarial: 5.79/10). - Adversarial questions are hardest for all models; 7B models score near 0 (Maya: 0.16, Chitrarth-1: 0.03). - GPT-4o lags behind LLaMA-4 and Gemma-3 on multilingual VQA despite strong English performance.

VQA (Multilingual/Adversarial): - Gemini-2.5 best across all languages on adversarial (e.g., Punjabi: 5.71, English: 5.79); GPT-4o second in most cases but scores collapse in Kannada (0.67) and Tamil (1.70). - All models score substantially below their English performance in low-resource languages.

MMT: - Gemini-2.5 dominates (BLEU: 44.51 Bengali, 53.27 Gujarati, 52.39 Punjabi). - LLaMA-4 and Gemma-3 competitive for second place; Malayalam most challenging for all models. - Chitranuvad (fine-tuned on Visual Genome for 3 languages) beats its base Chitrarth-1 in Hindi, Kannada, Malayalam, Telugu but surprisingly underperforms in Bengali.

OCR (ANLS word-level, selected): - Gemini-2.5 best overall; Malayalam highest scoring language (59.64), others remain challenging (Marathi: 8.06, Hindi: 16.01). - GPT-4o performs poorly: Malayalam word-ANLS 94.67 (higher = worse; ANLS is a similarity measure, so these figures indicate error rates), Gujarati 90.54. - Surya ranks second in Gujarati, Hindi, Kannada, Marathi; Chitrapathak second in Bengali, Tamil, Telugu.

Image necessity: Removing visual input causes consistent accuracy drops; short-answer tasks most affected (e.g., Gemini-2.5 English: 4.14 → 8.22 with image).

Limitations

• Only 106 images in the VQA-Parallel and MMT tracks, limiting cross-lingual analysis statistical power.
• QA generation relies on Gemini models, introducing potential systematic biases in question style and cultural framing.
• States are used as proxies for cultural groups, which conflates intra-state diversity.
• OCR corpus drawn exclusively from Wikisource, skewing toward formal/literary text and excluding scene text or informal handwriting.
• LLM-as-a-Judge evaluation for open-ended questions may not capture all dimensions of cultural correctness.
• No fine-tuned baselines specifically targeting the benchmark; results represent zero-shot generalization only.

Relevance to Vision-Language Models

IndicVisionBench directly exposes a systematic failure mode in current VLMs: strong aggregate benchmark performance does not transfer to culturally specific or low-resource-language settings, with even frontier models (Gemini-2.5, GPT-4o) scoring near-zero on adversarial cultural questions in several Indic languages. For researchers tracking VLMs, this benchmark provides a rigorous, multi-task evaluation harness that decomposes multimodal competence into recognition, reasoning, translation, and script-level OCR, enabling fine-grained diagnosis of where cultural and linguistic generalization breaks down. It complements Western-centric benchmarks like MMBench and MMStar by surfacing underrepresented failure modes—adversarial cultural reasoning and multilingual OCR—that existing suites do not probe. The paired parallel corpus also enables controlled cross-lingual transfer studies, directly relevant to multilingual VLM training and alignment research.