Skip to content

Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Kaleidoscope: massively multilingual in-language VLM evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Kaleidoscope is a large-scale, in-language multimodal benchmark of 20,911 multiple-choice exam questions spanning 18 languages and 14 subjects, built via open-science collaboration with native speakers. Unlike prior multilingual benchmarks relying on English translation, it sources questions directly from real-world national exams to preserve cultural and linguistic authenticity. Evaluation of top VLMs reveals systematic weaknesses in low-resource languages, non-Latin scripts, STEM reasoning, and complex visual inputs like diagrams and formulas.

Problem

Existing VLM evaluations are predominantly English- and Western-centric; multilingual benchmarks that exist largely rely on machine translation of English datasets, which introduces translationese artifacts and fails to capture culturally grounded knowledge. There is no large-scale benchmark that simultaneously covers diverse languages and diverse visual modalities at the intersection of multilinguality and multimodality.

Method

Kaleidoscope is constructed through a participatory open-science pipeline involving contributors from 20 nations. Questions are sourced directly from official government exam repositories, teacher-authored question banks, and educational databases in their original languages. Processing involves automated PDF/OCR extraction (Mathpix, GPT-4o) followed by two-stage manual validation by independent annotators. Each question is formatted as 4-option MCQA with 17 metadata fields including language, subject, educational level, image type (diagram, graph, table, photo, formula, map, figure, text-rich), and license. Evaluation uses zero-shot CoT prompting (closed models, in-language instructions) and direct JSON-structured answer generation (open-weight models, English instructions); accuracy is macro-averaged equally across all 18 languages.

Key Contributions

  • Largest multilingual multimodal exam benchmark: 20,911 questions, 18 languages across 8 language families, 14 subjects from 6 broad domains
  • 55% of questions (11,459) require image understanding; remaining 45% are text-only, enabling modality-controlled analysis
  • In-language, native-authored questions with rich per-question metadata enabling fine-grained diagnostic analysis
  • Systematic evaluation of 11 VLMs spanning 3B–72B parameters (Qwen2.5-VL, Aya-Vision, Molmo, Pangea, Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro)
  • Quantification of modality gap, script bias, language-resource gap, and STEM vs. Humanities performance disparities

Results

  • Best overall accuracy: Claude 3.5 Sonnet (62.91%), Gemini 1.5 Pro (62.10%), GPT-4o (58.32% raw / 62.10% valid)
  • Modality gap: GPT-4o drops 21.6% from text-only to multimodal; Molmo drops only 3.69%; all closed models show larger gaps than open-weight models
  • Image type hierarchy: text-rich images highest (GPT-4o: 86.2%), photos high (81.5%), tables (76.5%), then diagrams lowest (62.9% best); formulas also weak (Claude: 52.1%, best among closed)
  • Domain gap: humanities avg. best accuracy 83.7% vs. STEM 59.2%; closed models reach 93.4% in Sociology but fall below 50% in Mathematics/Physics/Engineering
  • Language/script gap: all models consistently above parity line for Latin vs. non-Latin scripts; Nepali and Telugu weakest; Lithuanian highest (confounded by subject composition)
  • Best open-weight: Qwen2.5-VL-72B at 52.94% overall; Qwen2.5-VL-7B leads all 7–8B models
  • Image ablation (Qwen2.5-VL-7B): standard multimodal 36.85%, no-image 33.44% (−3.41%), random-image 32.56% — confirms non-trivial but limited visual grounding
  • Scaling: consistent gains across Qwen2.5-VL 3B→72B

Limitations

  • Question counts per language vary widely (126 for Nepali to 2,000 for Portuguese/Serbian/Persian), creating uneven statistical power
  • Some languages (Nepali, Telugu) are 100% multimodal, conflating resource-level effects with modality difficulty
  • Open-weight models evaluated with English-only instructions regardless of question language, potentially underestimating multilingual instruction-following ability
  • Lithuanian's anomalously high performance is attributed to subject composition rather than genuine language capability, suggesting subject-language confound
  • No adversarial or free-response evaluation; MCQA format allows above-chance text-only guessing (Qwen2.5-VL-7B: 33.45% no-image vs. 25% random baseline)
  • Benchmark covers 18 languages; many more remain unevaluated

Relevance to Vision-Language Models

Kaleidoscope directly exposes a critical blind spot in VLM development: strong English multimodal performance does not transfer proportionally to other languages, particularly those using non-Latin scripts or low training data. The modality gap finding — that larger closed models paradoxically show greater degradation on multimodal vs. text-only inputs than smaller open-weight models — challenges the assumption that scale uniformly improves multimodal robustness. The STEM–Humanities gap and image-type hierarchy results provide a concrete diagnostic map for where current VLM visual reasoning remains shallow (symbolic/diagrammatic reasoning) versus mature (photos, text-in-image). For researchers tracking VLMs, this benchmark fills a necessary role as a held-out, culturally grounded test suite that is resistant to contamination via English-centric pretraining.