Skip to content

ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Linjie Li (Microsoft); ICLR 2026; multimodal interleaved chain-of-thought emergent properties

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ThinkMorph is a unified VLM fine-tuned on ~24K interleaved text–image reasoning traces across four vision-centric tasks, treating text and image thoughts as complementary rather than isomorphic modalities. It achieves an average 34.7% gain over its base model on vision-centric benchmarks and generalizes to out-of-domain tasks, matching or outperforming larger proprietary VLMs. Fine-tuning also unlocks emergent behaviors—unseen visual manipulations, autonomous reasoning-mode switching, and improved test-time scaling—not explicitly present in training.

Problem

Existing multimodal chain-of-thought approaches either rely on brittle external visual tools or produce interleaved reasoning where text and image tokens are isomorphic (conveying the same information in different modalities), providing no complementary signal. Pure text CoT fails on vision-centric tasks requiring spatial manipulation, fine-grained grounding, or iterative visual verification. There is no principled, generalizable recipe for constructing mutually reinforcing interleaved text–image reasoning traces.

Method

ThinkMorph fine-tunes Bagel-7B (a unified text+image generation model) on 24,990 curated interleaved reasoning traces spanning four tasks with increasing visual engagement:

  1. Jigsaw Assembly — text tokens describe puzzle pieces; image tokens visualize re-arranged hypotheses; final text verifies spatial coherence.
  2. Spatial Navigation — text establishes a coarse path plan; image tokens render the trajectory overlay on the maze; text verifies move sequences.
  3. Visual Search — text hypothesizes the region of interest; image tokens draw a bounding box; text confirms object attributes.
  4. Chart Refocus — text identifies relevant data elements; image tokens highlight chart regions; text extracts values and computes answers.

The thought sequence is formalized as \(\mathcal{T} = (\hat{m}_1, \hat{m}_2, \ldots, \hat{m}_n)\) where \(\hat{m}_i \in \{\hat{t}_i, \hat{v}_i\}\) and modality transitions are controlled via <image_start> / <image_end> delimiter tokens. Training uses a dual objective: MSE loss \(\mathcal{L}_\text{img}\) on image tokens and cross-entropy loss \(\mathcal{L}_\text{text}\) on text tokens. Unimodal baselines (text-only via GPT-4.1 prompting; visual-only using image outputs from the interleaved traces) are derived from the same data for controlled comparison.

Key Contributions

  • First systematic study of when and how interleaved multimodal reasoning surpasses text-only and visual-only modes, using controlled ablations across four tasks.
  • A curated 24K-trace dataset with task-specific visual manipulations (bounding boxes, path overlays, piece re-arrangement, chart highlights) that enforce complementarity between modalities.
  • Identification of three emergent properties: (1) Unseen Visual Manipulations (zoom-in, inpainting, multi-box, motion forecasting, perspective shift, cropping — up to 10% of inference operations); (2) Autonomous Mode Switching (5.3% of inferences switch to text-only despite interleaved-only training, gaining +7.29% on those instances); (3) Better Test-Time Scaling via Diversified Thoughts (+8.0% on Jigsaw Assembly with more samples).
  • Demonstration that pretraining supplies latent visual manipulation skills while interleaved fine-tuning directs them toward structured reasoning steps.

Results

  • Average gain over base Bagel-7B on vision-centric tasks: +34.74% (interleaved vs. base).
  • Spatial Navigation (VSP): base 0.83% → interleaved 86.67% (+85.84%); text-only 49.17%; visual-only 85.50%.
  • Jigsaw Assembly (VisPuzzle): base 35.00% → interleaved 73.75% (+38.75%); BLINK-J out-of-domain: +6.00% over text-only.
  • Visual Search (VStar, OOD): interleaved 63.87% vs. text-only 56.02% vs. visual-only 58.63%.
  • MMVP (OOD): interleaved 80.33% vs. Gemini 2.5 Flash ~80.33% (matched); base 70.33%.
  • SAT spatial reasoning (OOD): ThinkMorph 52.67% vs. InternVL3.5-38B 49.33%.
  • ChartQA (in-domain): text-only slightly outperforms interleaved (+1.88%), showing interleaving is not universally necessary.
  • Interleaved outperforms the next-best unimodal mode by 5.33% averaged across vision-centric tasks.

Limitations

  • Training scale is modest (~24K traces); generalization breadth may be constrained by the four task types.
  • Emergent manipulations (up to 10% of operations) are not explicitly supervised; their reliability and failure modes are not fully characterized.
  • The paper is truncated before full reporting of test-time scaling and mode-switching analysis, so quantitative bounds on Property 3 are incomplete in the provided text.
  • Relies on Bagel's large-scale multimodal pretraining for latent manipulation skills; the approach may not transfer to base models without strong visual generation pretraining.
  • GPT-5 used as LLM-as-a-Judge for open-ended evaluation, introducing external model dependence.
  • No analysis of failure cases where inappropriate mode switching degrades performance.

Relevance to Vision-Language Models

ThinkMorph directly advances the frontier of interleaved multimodal reasoning in VLMs by establishing that text and image tokens in a CoT trace must be complementary (not redundant), which is a principled design criterion largely absent from prior work. The emergent visual manipulation behaviors—especially unseen operations like zoom-in and inpainting arising without explicit supervision—suggest that unified VLMs with strong visual generation pretraining have latent capabilities activatable through structured fine-tuning. The autonomous mode-switching finding has direct implications for inference efficiency in VLMs: models can self-regulate between text-only and interleaved reasoning, reducing unnecessary image generation cost. For researchers tracking VLMs, ThinkMorph offers a controlled experimental framework and a dataset for studying when visual CoT genuinely helps versus when text CoT suffices—a question central to designing next-generation multimodal reasoning systems.