Skip to content

Vision Language Models are Biased

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; systematic evidence that VLMs exhibit strong prior-knowledge biases

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

State-of-the-art VLMs default to memorized prior knowledge instead of visually analyzing counterfactual images, achieving only 17.05% mean counting accuracy across 7 domains when signature elements are subtly modified. The paper introduces VLMBias, a semi-automated benchmark with 1,392 human-verified counterfactual images spanning animals, logos, flags, chess, game boards, optical illusions, and patterned grids. Removing image backgrounds nearly doubles accuracy (+21.09 points), revealing that background visual context is a primary trigger for biased responses.

Problem

Existing VLM bias benchmarks rely on adversarially worded prompts (e.g., "Is the mouse smaller than the cat?") that conflate language bias with visual bias, making it unclear whether VLM failures stem from the text prompt or the image. No prior work tested whether VLMs can count or identify elements in counterfactual images using neutral, objective prompts — precisely the regime where strong prior knowledge from pretraining is most dangerous.

Method

VLMBias benchmark construction: For each of 7 domains, the authors systematically modify a single signature visual attribute of a well-known subject (e.g., adding a stripe to a 3-stripe Adidas logo, adding a leg to a 4-legged mammal) to produce counterfactual (CF) images. Photo-realistic domains (animals, logos) use SOTA text-to-image editors (Gemini-2.0 Flash, GPT-4o); abstract domains (flags, chess, game boards, patterned grids) use programmatic generation. All images are human-reviewed to reject low-quality or ambiguous cases.

Each CF image is tested at three resolutions (\(D \in \{384, 768, 1152\}\)) with two neutral prompt phrasings. Three question types are asked per image: - Q1/Q2: Direct and rephrased counting questions ("How many stripes…?", "Count the stripes…") - Q3: Y/N identification ("Is this the Adidas logo?") to disentangle counting ability from visual detection

Bias rate is defined as the frequency with which a VLM's answer matches the predefine biased response (i.e., the canonical real-world count), which is incorrect for the CF image.

Novel patterned grids test intra-image bias: \(G \times G\) grids (\(G \in \{6,\ldots,12\}\)) follow a symmetric count pattern, with one anomaly cell modified. This isolates bias toward dominant internal image patterns rather than external memorized knowledge.

Five VLMs are evaluated: Gemini-2.5 Pro, Claude Sonnet-3.7 (non-thinking), GPT-4.1 (non-thinking), o3, o4-mini.

Key Contributions

  • VLMBias benchmark: 1,392 human-verified CF images across 7 domains; neutral counting/identification prompts with no adversarial text injection
  • Bias rate metric: Quantifies the proportion of responses matching the memorized prior, separating bias-driven errors from general counting failures
  • Sanity-check protocol: Confirms 100% VLM accuracy on unmodified images before attributing CF failures to bias
  • Background masking experiment: Demonstrates that occlusion of background pixels nearly doubles accuracy (+21.09 points), isolating contextual visual cues as a trigger
  • In-image text injection experiment: Adding brand names (e.g., "Adidas") directly onto CF images drops accuracy by a further 2–6 points, confirming text-corpus bias
  • Novel patterned grids domain: Introduces a visual stimulus with no prior Internet existence to isolate intra-image pattern bias
  • Overthinking analysis: Shows a non-monotonic relationship between reasoning tokens and accuracy, peaking near 40% before declining

Results

  • Overall mean accuracy on CF images: 17.05% across 5 VLMs and 7 domains; best single model is o4-mini at 20.25%
  • Sanity check: All 5 VLMs score 100% on unmodified images across all tasks
  • Domain breakdown (mean over 5 VLMs):
  • Animals: 2.12% (birds: 1.01%, mammals: 2.50%)
  • Logos: ~6.13% (car logos: 0.44%, shoe logos: 17.57%)
  • Flags: ~9.25% (stars: 11.79%, stripes: 4.52%)
  • Chess pieces: 26.25% (thinking VLMs >26% vs. non-thinking <10%)
  • Game boards: 2.26% (0% on Sudoku and Go)
  • Optical illusions: 50.87% (≈ random chance); bias rate 78.02% on CF versions
  • Patterned grids: 22.44%; 43.45% of errors match the surrounding cell pattern
  • Bias rate overall: VLMs produce biased answers 75.70% of the time
  • Background removal: Accuracy increases by +21.09 points
  • In-image label injection: Accuracy decreases by 2–6 points
  • Thinking tokens: Accuracy peaks near ~40% then declines with additional reasoning tokens; non-thinking and thinking models perform comparably overall

Limitations

  • Dataset size per domain is modest (e.g., 91 animals, 207 logo images, 84 game board images), limiting statistical power for fine-grained cross-model comparisons
  • CF images are generated by some of the same model families used for evaluation (Gemini, GPT), introducing potential generative bias despite cross-family controls
  • The benchmark is restricted to counting and Y/N identification; it does not cover other objective visual tasks (localization, depth estimation, etc.)
  • Patterned grids lack natural ecological validity as a visual domain
  • The overthinking analysis is conducted on a subset of questions, limiting generalizability of the token-accuracy curve
  • Models with tool use (e.g., image cropping/zooming) are not systematically evaluated; the authors note VLMs underutilize pointing and cropping tools due to overconfidence

Relevance to Vision-Language Models

This paper directly characterizes a fundamental failure mode of current VLMs: the dominance of text-corpus priors over visual evidence in objective, counterfactual scenarios. For practitioners tracking VLMs, it challenges the assumption that increased chain-of-thought reasoning mitigates hallucination — in fact, overthinking worsens bias-driven errors beyond a threshold. The background-masking result is particularly actionable, suggesting that VLM visual attention is strongly modulated by global scene context rather than the locally queried object. The work complements prior hallucination and counting benchmarks (HallusionBench, BlindTest) by using neutral prompts and an automated CF generation pipeline, establishing a cleaner causal attribution of errors to memorized priors.