Skip to content

Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PANGEA is a fully open-source multilingual multimodal LLM trained on PANGEA-INS, a 6M-sample instruction dataset spanning 39 languages that combines machine-translated English instructions with a curated multicultural understanding corpus. It outperforms all open-source MLLMs on multilingual tasks by an average of 10.9% while maintaining competitive English performance, and ships with PANGEA-BENCH, a new 14-dataset, 47-language evaluation suite.

Problem

Existing MLLMs are trained predominantly on English- and Western-centric data, causing (a) degraded multilingual performance and higher tokenization costs for non-English languages, (b) outputs misaligned with non-Western socio-cultural norms, and (c) failure to recognize geographically diverse visual content. Prior multilingual MLLM efforts (PALO, mBLIP) still exhibit pronounced cross-language performance inequality and lack rigorous cultural understanding evaluation.

Method

PANGEA uses LLaVA-Next architecture with Qwen2-7B-Instruct as the LLM backbone and CLIP-ViT-Large-Patch14-336 as the vision encoder. Training follows a two-stage pipeline: (1) vision-language connector pretraining on LLaVA-LCS-558K, then (2) full fine-tuning on PANGEA-INS for 1 epoch (1344 H100 GPU-hours).

PANGEA-INS (6M samples, 39 languages, 40% English / 60% multilingual) is built via three strategies: - Machine translation: High-quality English instruction pool (ALLaVA, Cambrian, ShareGPT-4o, math/code data) translated with Gemini 1.5 Pro (preferred over NLLB-3B due to better handling of code and complex instructions), followed by automated post-processing to fix mismatched conversation turns. - Multicultural instruction pipeline: 10M images sampled from LAION-Multi undergo heuristic filtering (CLIP score, NSFW, dedup) then LLM scoring with Llama-3.1-8B for text quality, subject classification, and country/region classification. ~1M culturally specific images are retained, recaptioned in the language of cultural origin using Gemini 1.5 Pro with alt-text grounding, then turned into QA pairs across 13 task types (Information Seeking, Cultural Interpretation, Critical Reasoning, etc.). - Existing multilingual datasets: Integration of open-source multilingual datasets (Russian GQA, Vietnamese Doc/OCR VQA, French Doc/Table-VQA, MTVQA, STAIR Captions, etc.).

PANGEA-BENCH provides zero-shot evaluation across 5 multimodal tasks (multimodal chat via xChatBench + M-LlavaBench, captioning via XM100, cultural understanding via CVQA + MaRVL, multilingual VQA via xGQA + MaXM, multi-subject reasoning via xMMMU + M3Exam) and 3 text-only tasks (TyDiQA, FLORES-Sub, MMMLU/XStoryCloze/MGSM). xChatBench uses human-annotated rubrics scored by an LLM judge on a 1–5 scale for finer-grained evaluation than prior coarse-grained methods.

Key Contributions

  • PANGEA-INS: 6M multilingual multimodal instruction dataset across 39 typologically diverse languages, fully open-sourced, with 1.5M culturally specific samples from a novel multicultural pipeline.
  • PANGEA-7B: Fully open-source (data + code + checkpoints) multilingual MLLM that outperforms all open-source models in multilingual settings while matching SoTA on English.
  • PANGEA-BENCH: Holistic 14-dataset, 47-language evaluation suite including two newly introduced benchmarks (xChatBench with fine-grained rubric-based scoring; xMMMU, a 7-language translated MMMU).
  • Ablation insights: Empirical characterization of English data proportion optimum (~40%), scaling effect of instruction quantity, and the relationship between per-language training sample proportion and downstream performance (near-linear for high-resource, disproportionate gains for low-resource).

Results

  • PANGEA-7B surpasses the best open-source model by +0.4% on English and +10.9% on multilingual tasks averaged across PANGEA-BENCH.
  • xChatBench (multilingual): 35.8 vs. 28.5 (Llava-OV-7B next best); MaRVL: 57.2 vs. 53.7 (Llama3.2-11B); xGQA (multilingual): 60.2 vs. 48.2 (Llava-OV-7B); MaXM (multilingual): 53.3 vs. 43.9 (Llama3.2-11B).
  • On text-only tasks, PANGEA-7B (72.8 en / 54.3 mul) outperforms all multimodal LLM baselines and approaches its text backbone Qwen2-7B-Instruct (66.6 / 54.5), with notable gains on MGSM (82.0 en vs. 48.8 for Qwen2, attributed to math instruction data).
  • Matches or exceeds Gemini-1.5-Pro and GPT-4o on xGQA but trails them on multimodal chat and complex reasoning tasks.
  • Optimal English-to-multilingual data ratio found to be 40:60; multilingual performance peaks at 40% English and drops sharply at 100% English or 0% English.

Limitations

  • Performance gaps remain compared to proprietary models (GPT-4o, Gemini-1.5-Pro), particularly in multimodal chat and complex reasoning.
  • Machine translation still introduces errors in nuanced contexts and cultural subtleties; translation quality with Gemini 1.5 Pro is satisfactory but not equivalent to human annotation.
  • LAION-Multi images, even after filtering, are not always culturally representative of the speaking population; ~30% of curated images were lost to accessibility/download issues.
  • Multilingual OCR remains a challenge (discussed in appendix but not fully solved).
  • Low-resource language support is uneven; performance still lags behind high-resource languages despite disproportionate gains from small training increases.
  • xMMMU is machine-translated (GPT-4o), introducing potential translation artifacts into the reasoning benchmark.

Relevance to Vision-Language Models

PANGEA directly advances the frontier of multilingual VLMs by demonstrating that a 7B-parameter open model can close the gap with proprietary systems on non-English tasks when training data combines high-quality translated instructions with culturally grounded multimodal data. The PANGEA-INS curation pipeline — particularly the LLM-scored cultural image filtering and rubric-based instruction generation — offers a replicable methodology for addressing Western-centric bias in VLM training data. PANGEA-BENCH fills a critical evaluation gap by providing standardized, culturally sensitive multilingual benchmarks (xChatBench, xMMMU) that expose weaknesses invisible to English-only evaluation. The English-data proportion ablations provide practical guidance for anyone training multilingual VLMs under constrained budgets.