Skip to content

Caption This, Reason That: VLMs Caught in the Middle

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper evaluates state-of-the-art VLMs through a cognitive science lens—Perception, Attention, and Memory (PAM)—using procedurally generated benchmarks. It finds that spatial reasoning and selective attention are persistent failure modes across all tested models, and identifies the core bottleneck as deficient integration of visual features into the reasoning chain rather than perceptual failure per se. A simple self-captioning (vision-text decoupling) intervention substantially recovers performance, especially on location-based tasks.

Problem

Existing VLM benchmarks (MMMU-Pro, MMBench, MME) evaluate high-level cognitive reasoning but do not isolate the foundational cognitive axes—Perception, Attention, and Memory—that underlie them. Without decomposing failures along these axes, it is unclear whether VLM deficits arise from perceptual encoding, information maintenance, selective attention, or downstream reasoning integration.

Method

The authors construct two datasets using the iWISDM procedural generation environment with ShapeNet objects (8 categories, 8 objects each, 4 spatial locations): - PAM dataset: 22 tasks explicitly targeting Perception (object category/location identification), Attention (spatial- and feature-cued selection from distractors), and Memory (recall after blank/distractor frames), each in Report and Compare variants. - CVR dataset: Composite tasks of Low/Medium/High complexity requiring combinations of the above abilities.

Seven VLMs are evaluated: GPT-4o, GPT-4o-Mini, Qwen2.5-VL-7B/72B, InternVL2.5-8B, LLaVA-OneVision-7B, MiniCPM-V-2.6-8B.

To diagnose the reasoning bottleneck, three vision-text decoupling conditions are applied to Qwen2.5-VL-7B: - PC (Pre-captioned): ground-truth captions replace images. - SC (Self-captioning): model's own captions replace images in a separate pass. - SC-I (Self-captioning Interleaved): model captions are inserted between images.

LoRA fine-tuning of Qwen2.5-VL-7B on CVR tasks (1k/10k/100k trials, dropout 0.2) is evaluated for generalization to PAM and out-of-distribution benchmarks.

Key Contributions

  • Introduction of the PAM benchmark—a procedurally generated, cognitively grounded evaluation suite decomposing VLM ability into Perception, Attention, and Memory with controlled stimuli.
  • Identification of spatial reasoning and selective attention as the primary failure modes across all tested VLMs, including GPT-4o.
  • Vision-text decoupling analysis showing that the bottleneck lies in visual-feature-to-reasoning integration, not in perceptual encoding or LLM reasoning capacity per se—demonstrated by large SC gains (+18.3% avg on location tasks vs. +3.5% on category tasks for Qwen2.5-VL-7B).
  • Demonstration that LoRA fine-tuning on as few as 1k CVR trials yields significant PAM improvements, with 100k trials providing +34.3 pp on Perception-Loc and +44.3 pp on CVR-Loc-H.
  • Validation that PAM/CVR accuracy strongly correlates with MMMU-Pro and VQAv2 performance, confirming diagnostic validity of the benchmark.

Results

  • GPT-4o achieves closest-to-human performance but still lags significantly on Spatial Attention (75.93% vs. 98.75% human) and Memory-Loc (83.47% vs. 96.25% human).
  • Open-source 7–8B models lag substantially on location tasks: Qwen2.5-VL-7B scores 44.33% Perception-Loc vs. 97.50% human; Spatial Attention 57.48% vs. 98.75%.
  • Self-captioning (SC) on Qwen2.5-VL-7B yields +28.67 pp on Perception-Loc, +22.00 pp on Feature Attention, +18.15 pp on Spatial Attention, +26.89 pp on Memory-Loc (all p<0.001); CVR-Loc-H improves +17.33 pp (p<0.05).
  • Pre-captioning (PC) produces largest gains (+55.00 pp Perception-Loc, +47.39 pp Memory-Loc), bounding the possible improvement if captions were perfect.
  • LoRA 100k improves Qwen2.5-VL-7B by +34.33 pp Perception-Loc, +27.69 pp Memory-Loc, +44.33 pp CVR-Loc-H; modest gains on MMBench/VQAv2, slight decrease on MMMU-Pro at smaller data sizes.
  • PAM Memory↔CVR correlation: R²=0.94; Attention↔CVR R²=0.82; Perception↔CVR R²=0.59.

Limitations

  • Synthetic stimuli: only 8 ShapeNet object categories on uniform backgrounds, limiting ecological validity.
  • Fine-tuning restricted to Qwen2.5-VL series (7B and 32B); broader model coverage is absent.
  • SC method approximately doubles inference runtime, a practical constraint for deployment.
  • Fine-tuning used randomly distributed complexity; complexity-balanced curricula were not explored.
  • MMMU-Pro performance slightly decreases with small LoRA datasets, indicating limited out-of-distribution generalization.
  • Human baseline collected for PAM/CVR but not at scale; human variance is high on complex CVR tasks.

Relevance to Vision-Language Models

This paper directly addresses a core open question in VLM research: where does visual reasoning break down? By borrowing the Perception-Attention-Memory taxonomy from cognitive science, it provides a principled diagnostic framework that goes beyond aggregate benchmark scores and reveals that spatial reasoning failures are primarily a Chain-of-Thought integration failure rather than a perceptual one—an actionable finding for training and prompting strategies. The vision-text decoupling result is particularly consequential: it shows that self-captioning as an intermediate reasoning step can serve as a cheap but effective CoT scaffold, relevant to any researcher working on VLM reasoning, spatial understanding, or structured prompting. The strong correlation between PAM/CVR and established benchmarks also positions this suite as a lightweight, interpretable diagnostic complement to MMMU-Pro and VQAv2.