Skip to content

See It from My Perspective: How Language Affects Cultural Bias in Image Understanding

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs exhibit a systematic Western bias in both objective and subjective image understanding tasks, performing consistently better on Western examples than East Asian ones. The authors trace this bias to the language mix used during text-only LLM pre-training, showing that training on more Chinese text reduces Western bias even when prompting in English. Prompting in a culturally closer language helps, but only substantively when that language was prominent in pre-training.

Problem

VLMs are multilingual but not multicultural: they perform worse on East Asian images/annotations across objective (object identification, VQA) and subjective (art emotion classification) tasks. Prior work attributed Western bias to image distribution in multimodal corpora; this paper isolates the separate, underexplored role of the language distribution during text-only LLM pre-training.

Method

Two-step design using culturally stratified splits of Dollar Street (object identification), A-OKVQA/CVQA (VQA), and ArtELingo (art emotion classification), each partitioned into Western and East Asian subsets by image origin and annotator language.

Step 1 (Bias Characterization): Seven off-the-shelf VLMs (BLIP2, LLaVA-1.5 variants, Qwen-VL, mBLIP) evaluated zero-shot in English and Chinese; Western bias measured as the ratio of Western to East Asian performance (bias = 1 means unbiased).

Step 2 (Bias Sourcing): Six controlled mLLaVA variants trained by fusing CLIP with two 7B instruction-tuned LLMs—Llama2 (predominantly English, 2T tokens) vs. Baichuan2 (balanced English/Chinese, 2T tokens)—across three fusion corpus conditions (English-only, Chinese-only, bilingual). This 2×3 factorial isolates the contribution of (A) LLM pre-training language mix, (B) prompting language, and (C) fusion corpus language mix. Hidden-state decoding via logit lens probes internal language representation.

Key Contributions

  • Demonstrates that 90%+ of evaluated off-the-shelf VLMs exhibit Western bias on both objective and subjective tasks.
  • Causally links Western bias to LLM pre-training language mix: Baichuan2-based mLLaVA consistently reduces bias vs. Llama2-based variants.
  • Shows that Chinese pre-training reduces bias even when prompting in English, especially on subjective tasks (ArtELingo bias: 3.45→2.71 on opposite-valence subset).
  • Establishes that culturally closer prompting language yields much larger bias reductions when that language was well-represented in pre-training (Baichuan2 + Chinese prompting reduces ArtELingo opposite-valence bias from 2.72→1.23 vs. Llama2's 3.45→2.47).
  • Shows via logit lens that Llama2 processes all input internally in English even when generating Chinese, while Baichuan2 activates genuinely bilingual representations.
  • Demonstrates that a bilingual multimodal fusion corpus cannot substitute for multilingual text-only pre-training.

Results

  • Off-the-shelf VLMs: 90%+ exhibit Western bias across all three tasks; Chinese prompting reduces bias in only 15/30 model-task evaluations.
  • VQA (LLM pre-training effect, Chinese prompting): Baichuan2 vs. Llama2 reduces bias from 1.50→1.17 (en/zh subset) and 1.29→1.16 (regional subset).
  • ArtELingo opposite valence (LLM pre-training effect, Chinese prompting): 2.47→1.23 (Baichuan2) vs. weaker reduction with Llama2.
  • ArtELingo all (LLM pre-training effect, English prompting): 3.45→2.71 (opposite valence) and 2.20→1.98, confirming pre-training effect transfers to English inference.
  • Fusion corpus: Bilingual fusion shows no consistent benefit for objective tasks; on subjective tasks it strengthens prompt-language/cultural-perspective alignment but is secondary to pre-training language mix.
  • Logit lens: Baichuan2 decoded hidden states are closer to Chinese annotator rationales in ArtELingo than Llama2 hidden states.

Limitations

  • Controlled study restricted to English and Chinese due to lack of comparable multilingual LLMs; results for other non-Western languages remain untested.
  • East Asia is the highest-resourced non-Western region; measured bias is a lower bound—bias relative to less-resourced cultures is likely worse.
  • Cultural proxy (country of image origin / annotator language) is a simplification that cannot capture intra-cultural variation.
  • Benchmarks used for Western splits (MS-COCO, A-OKVQA) are themselves skewed Western, potentially underestimating true bias magnitude.
  • Study does not scale to very large models; findings are on 7B-parameter variants only.

Relevance to Vision-Language Models

This paper directly challenges the assumption that multilingual VLMs are implicitly multicultural, revealing that language choice during text-only LLM pre-training—not just vision encoder training data or multimodal fusion corpora—is a primary driver of cultural bias in image understanding. For researchers tracking VLMs, this has immediate architectural implications: building equitable multimodal models requires diverse multilingual foundation LLMs, not just diverse image datasets or prompting strategies. The logit lens finding that Llama2 "thinks in English" even when generating Chinese provides a mechanistic explanation for why translating prompts or captions post hoc fails to transfer cultural perspective, directly informing the design of future multilingual VLM training pipelines.