Skip to content

Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes "visual thoughts" as a unified explanatory framework for why multimodal chain-of-thought (MCoT) improves large vision-language models. Visual thoughts are intermediate reasoning steps that extract and cache task-relevant visual information from input images, enabling deeper cross-modal reasoning without reprocessing raw pixels at every step. Four distinct expression forms (Natural Language, Structured Language, Edited Image, Generative Image) are analyzed, revealing that clarity and conciseness—not image fidelity—determine effectiveness.

Problem

The two dominant MCoT paradigms—Textual-MCoT (T-MCoT, text-only rationales) and Interleaved-MCoT (I-MCoT, image-text interleaved rationales)—lack a unified mechanistic explanation for why they improve LVLM performance. Prior work debates which paradigm is superior without identifying what drives the improvements or providing principled guidance for choosing among them.

Method

The authors introduce visual thoughts as a unified concept: intermediate reasoning steps that distill instruction-relevant visual information into a cache-like representation, replacing repeated raw-image access during multi-step reasoning. They formalize visual thoughts via a conditional generation equation where downstream derivative reasoning steps attend to the visual thought cache rather than the original image.

Four expression categories are systematically evaluated across multiple LVLMs (LLaVA-1.5-7B/13B, Qwen2-VL-7B, GPT-4o-mini, GPT-4o) and benchmarks (MMVP, V*Bench, M³CoT, CoMT): 1. N-LANG (Natural Language): image captions generated before reasoning 2. S-LANG (Structured Language): JSON scene graphs 3. E-IMG (Edited Image): annotated/grounded images via external vision tools (Grounding DINO) 4. G-IMG (Generative Image): DALL-E 3–generated images conditioned on the query

To probe internal mechanisms, the authors analyze layer-wise attention distributions (how much attention each transformer layer pays to the original image vs. visual thoughts) and perform causal information-flow perturbation experiments (blocking information flow from visual thought or from image at specific transformer layers).

Key Contributions

  • First unified mechanistic account of MCoT: both T-MCoT and I-MCoT improve LVLMs specifically by introducing visual thoughts, not through modality-specific advantages per se.
  • Taxonomy of four visual thought expression strategies covering textual and visual modalities.
  • Empirical demonstration that clarity (Spearman ρ = 0.83, Pearson r = 0.85, P < 0.01) and conciseness (ρ = 0.72, r = 0.86, P < 0.01) of expression—not image fidelity (ρ < 0.15, P > 0.4)—drive MCoT performance gains.
  • Attention analysis showing visual thoughts sustain information propagation into deep transformer layers (past layer 12, where raw-image attention otherwise collapses to near zero).
  • Information-flow perturbation showing that blocking query→visual-thought flow before layer 19 causes task failure, while blocking query→image flow does not, confirming visual thoughts as the operative cache.

Results

  • All four visual thought strategies improve performance over w/o VT baseline across nearly all LVLMs and benchmarks (Table 1).
  • Image-form visual thought (E-IMG, G-IMG) outperforms text-form (N-LANG, S-LANG) on complex tasks; gap is largest on CoMT-Selection (47.83% improvement for E-IMG on GPT-4o).
  • N-LANG best on coarse-grained perception (MMVP); S-LANG best on relational reasoning (V*Bench-position); E-IMG best on attribute fine-graining (V*Bench-attributes); G-IMG best on multi-step reasoning (MÂłCoT).
  • Visual thoughts outperform caption-only baseline by >7% in complex scenes.
  • Reasoning cost: S-LANG averages 364 text tokens; E-IMG adds ~1,113 image tokens; G-IMG adds ~393 image tokens per inference.
  • In latter 50% of transformer layers, attention to visual thoughts exceeds attention to the original image across LLaVA-1.5-7B/13B and Qwen2.5-VL-3B/7B; attention gain increases with model depth (architecture matters more than parameter count).

Limitations

  • Only single-round visual thought interactions are studied; multi-round iterative visual thought generation is left to future work.
  • G-IMG uses DALL-E 3 as a proxy because direct Any-to-Any LVLM generation produces poor logic quality, introducing an external dependency.
  • External tool noise (from Grounding DINO, Stable Diffusion) in I-MCoT degrades performance and can fall below direct inference when noise is excessive.
  • Human annotation is used for clarity/conciseness/fidelity scoring, introducing subjectivity and limited scalability.

Relevance to Vision-Language Models

This paper directly addresses a foundational question in LVLM reasoning: what actually makes MCoT work? By identifying visual thoughts as the operative mechanism—a visual cache that mediates between raw pixels and linguistic reasoning—it provides a principled basis for designing or selecting MCoT strategies rather than relying on empirical trial-and-error. The finding that clarity and conciseness (not modality) drive effectiveness has practical implications for prompt design and training objectives in VLMs. The attention and information-flow analyses connect to broader mechanistic interpretability research in multimodal transformers, complementing work on visual grounding, cross-modal attention, and token efficiency in LVLMs.