Skip to content

VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a procedurally-generated benchmark that tests three forms of nonlocal visual reasoning—comparative perception, saccadic search, and smooth visual search—using synthetic, minimal-context tasks trivial for humans. Flagship VLMs (GPT-5, Gemini 2.5 Pro, Claude Sonnet 4) barely exceed random accuracy on the hardest variants, despite strong performance on existing visual benchmarks. The results reveal that current VLMs lack general-purpose visual algorithms and preferentially exploit language priors over direct image evidence.

Problem

Existing VLM benchmarks (ChartQA, AI2D, HallusionBench) either embed strong domain priors that models can exploit without true visual reasoning, or test perception in isolation. No prior work systematically evaluates sequential, evidence-chaining visual reasoning—tasks that require integrating information across multiple, possibly distant image regions rather than performing independent, parallel extractions from image to text space.

Method

Three procedural task generators produce synthetic image-question pairs with known-correct answers and human-trivial difficulty:

  1. Object Re-Identification (comparative perception): Two-image same/different judgment on composite geometric objects under rigid transformation. Three variants—Standard (contiguous shapes), Unconnected (non-contiguous), Pixel-Perfect (no transformation, pixel-identical positives)—isolate how Gestalt connectedness affects model behavior. Random baseline: 50%.

  2. Visual Scavenger Hunt (saccadic search): A labeled grid of colored shapes; the model must follow a written-label chain of length 2–4, with each step's target determined by the previous step's shape label. A decomposed single-step oracle experiment separates perceptual failure from chaining failure. Random baseline: ~9%.

  3. Circuit Connections (smooth visual search): A synthetic circuit diagram where the model must trace a wire from a breadboard port to its endpoint. Three variants—Standard (shared wire colors), Single Color (all wires identical), Unique Colors (each wire distinct)—control for color-cue shortcuts. Log-odds regression on Euclidean wire distance and number of crossings tests whether models behave like tracers. Random baseline: ~14.3%.

Models evaluated: GPT-5, Gemini 2.5 Pro, Claude Sonnet 4, o4-mini, o3, Claude 3.7 Sonnet, InternVL3-14B, Phi-4-Multimodal-14B, Qwen2.5-VL (7B/32B), Mistral Small 3.1-24B, Gemma 3-27B, Molmo-7B, Llama-3.2-Vision-11B. Few-shot setting; 200 or 125 examples per variant. Human baseline: 100% on Object Re-ID and Scavenger Hunt, 99.5% on Circuit Connections.

Key Contributions

  • Three procedural benchmark generators isolating comparative perception, saccadic search, and smooth visual search with synthetic, minimal-context stimuli.
  • Comprehensive evaluation of 13 closed- and open-source VLMs, including GPT-5, Gemini 2.5 Pro, and Claude Sonnet 4.
  • Variant-based ablation design that distinguishes perceptual failure from reasoning/chaining failure (e.g., single-step oracle decomposition for Scavenger Hunt; Single Color vs. Unique Colors for Circuit Connections).
  • Log-odds regression analysis showing that wire distance and crossing count predict model failure, inconsistent with genuine line-tracing behavior.
  • Evidence that strong models selectively engage visual attention (bypassing close inspection when Gestalt cues suggest similarity) and cannot self-correct from perceptual errors mid-chain.

Results

  • Object Re-Identification: No model significantly outperforms the 50% random baseline on the Standard variant. GPT-5 (77%), Claude Sonnet 4 (79%), Gemini 2.5 Pro (65%) improve on the Unconnected variant; all remain >20 pp below the human ceiling. Weaker models (Molmo-7B, Llama-3.2-11B) collapse to predicting a single class.
  • Visual Scavenger Hunt: Only GPT-5, o4-mini, and Gemini 2.5 Pro significantly beat the ~9% random baseline. o4-mini achieves 44.5% overall on 3-step chains vs. 90.67% on the decomposed single-step oracle—simple error multiplication predicts 83%, indicating the bottleneck is chaining, not perception. Remaining models perform at or near random.
  • Circuit Connections: Best single-color accuracy is 27% (Gemini 2.5 Pro) vs. 14.3% random. Top models drop sharply from Unique Colors (Gemini: 48%) to Single Color (27%), exposing reliance on color cues. Statistically significant negative distance effect across nearly all models (p ≤ 0.05); top models show significant negative crossing effects only in Single Color (e.g., Claude 3.7 Sonnet log-odds −0.93, p=0.01), consistent with choppy saccadic rather than smooth tracing.
  • No model approaches the human baseline on any task; Circuit Connections has the lowest performance ceiling overall.

Limitations

  • Benchmark uses only synthetic images; may not capture the full range of natural images where visual skims suffice.
  • Small evaluation sets (200 or 125 examples per variant) limit statistical power for fine-grained model comparisons.
  • Secondary metrics for o3 were lost due to a code bug and are excluded from F1 and log-odds analyses.
  • Evaluation covers at most 3 variants per task; other controlled manipulations (e.g., occlusion, scale extremes) are not tested.

Relevance to Vision-Language Models

This paper directly challenges the assumption that high scores on existing VLM benchmarks (ChartQA, AI2D) reflect robust visual processing, showing instead that models exploit language priors and domain conventions rather than executing general visual algorithms. The three-way taxonomy (comparative perception / saccadic search / smooth visual search) provides a principled framework grounded in human vision science (Treisman's FIT, Ullman's visual routines) for diagnosing architectural or training gaps in future VLMs. The finding that failures are hardest when tasks resist natural-language reformulation (smooth contour tracing) points to a fundamental limitation in how vision tokens are processed by the language decoder, echoing prior work suggesting perceptual deficits originate in the decoder rather than the image encoder. For researchers building or evaluating multimodal models, this benchmark suite offers diagnostic probes for nonlocal reasoning that existing suites miss.