Skip to content

Can We Talk Models Into Seeing the World Differently?

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs are significantly more shape-biased than their underlying vision encoders (e.g., CLIP), despite not reaching human levels (96%). Natural language prompts can steer this bias at inference time—particularly toward texture—without retraining, and this steerability generalizes beyond texture/shape to low/high-frequency biases.

Problem

It was unknown how the texture-vs-shape bias (well-studied in uni-modal vision models) behaves when a vision encoder is fused with an LLM, and whether language prompting can override the inductive visual biases inherited from the encoder—a prerequisite for lightweight, training-free alignment of model perception with human preferences.

Method

The authors evaluate 20+ VLMs (LLaVA variants, InternVL, InstructBLIP, Emu2, CogVLM, GPT-4V, Gemini Pro Vision, Qwen-VL) on the texture-shape cue-conflict dataset (Geirhos et al., 2019; 1,200 style-transferred ImageNet images with conflicting shape/texture labels from 16 classes). Shape bias is defined as Shape Accuracy / (Shape Accuracy + Texture Accuracy). Two evaluation tasks are used: VQA (forced single-letter response) and image captioning (embedding-based label extraction via ember-v1 + secondary LLM parsing). Bias steerability is tested via (1) hand-crafted shape/texture-biased prompts, (2) synonym substitution to confirm semantic grounding, and (3) automated LLM-as-optimizer prompt search maximizing/minimizing shape bias. A mechanistic analysis compares VLM decisions against isolated encoder zero-shot predictions using error consistency (Cohen's κ) and token-level logit confidence distributions. A new low/high-frequency cue-conflict dataset (1,200 samples, 30% LF + 70% HF blending) is also introduced.

Key Contributions

  • First large-scale study of texture/shape bias in LLM-based VLMs across VQA and image captioning.
  • Demonstrates that VLMs are systematically more shape-biased than their CLIP vision encoders, yet fall short of human shape bias (96%); most VLMs range 52.9–73.8% shape bias vs. 22% for ResNet-50 and ~60% for CLIP ViT-L/14.
  • Shows that LLMs convert flexible, dual-cue encoder representations into near-binary decisions (token confidence is almost binary), with ≥80% error consistency among VLMs.
  • Demonstrates prompt-based bias steering: shape bias can be shifted from ~49% to ~72% through prompting alone (23.3% delta for InternVL-Chat 1.1), with minimal accuracy loss.
  • Introduces a frequency-domain cue-conflict dataset and shows analogous prompt steerability for low/high-frequency bias.

Results

  • VLMs' shape bias: 52.9–73.8% (VQA), 54.1–73.2% (captioning), vs. humans at 96%, ResNet-50 at 22%, CLIP ViT-L/14 at ~60%.
  • VLMs show −9.5% to +7.4% shape-bias divergence from their encoders; error consistency with encoders is at most 78.7%, proving LLM processing meaningfully alters visual decisions.
  • Prompt steering (hand-crafted): neutral and shape-biased prompts behave similarly; texture-biased prompts reliably shift decisions toward texture without large accuracy drops (accuracy stays 80–91% across tested models).
  • Automated prompt search (InternVL-Chat 1.1): achieved 23.3% delta between extremes of shape bias; shape bias ranged ~49–72%.
  • Frequency bias steering: statistically significant changes achieved (e.g., InternVL-Chat 1.1 LF bias shifted from 34.5% neutral to 38.6% max / 32.9% min; p < 0.05).
  • GPT-4V outlier: ~10% refusal rate in VQA, 60.4% generic captions—anomalously poor cue-conflict accuracy.

Limitations

  • Cue-conflict dataset is small (1,280 images, 16 classes) and synthetic (style transfer); accuracy changes observed may not generalize to natural images.
  • Prompt engineering is a confound: different prompt choices could yield different bias measurements, and results are sensitive to the exact wording used.
  • Steering is partial—models cannot be pushed to fundamentally abandon their default bias; shape bias remains bounded even with automated search.
  • Model size does not clearly increase steerability (InternVL-Chat 1.2+ 34B vs. 13B showed no advantage).
  • GPT-4V and other closed-source models cannot be fully analyzed mechanistically (no logit access).
  • Findings may not generalize to radically different VLM architectures not covered in the study.

Relevance to Vision-Language Models

This work reveals a non-trivial and previously uncharacterized interaction between the language and vision components of VLMs: the LLM actively reshapes visual cue processing rather than passively forwarding encoder decisions, producing a qualitatively different shape/texture bias than seen in standalone encoders. For researchers tracking VLMs, this is directly relevant to understanding alignment, robustness, and the extent to which multi-modal fusion changes model perception. The prompt-steerability finding opens a low-cost path to bias correction at inference time, which is pertinent to work on VLM safety, hallucination, and distributional robustness. The mechanistic analysis of how LLMs collapse flexible encoder representations into binary decisions also connects to broader questions about information bottlenecks in cross-modal attention.