Skip to content

CoFFT: Chain of Foresight-Focus Thought for Visual Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoFFT is a training-free inference-time framework that improves VLM visual reasoning by emulating human visual cognition: iteratively generating diverse reasoning samples, selecting the best via joint visual-focus and reasoning-progression scoring, then zooming into the most relevant image region before the next reasoning step. Applied to Qwen2.5-VL, InternVL2.5, and LLaVA-Next, it yields consistent 3.1–5.8% average accuracy gains across seven benchmarks with lower compute cost than MCTS.

Problem

VLMs are easily distracted by visually dominant but semantically irrelevant image content, causing hallucinations and task-irrelevant reasoning. Existing approaches either fix visual focus to a single pass (DyFo, ICoT) or apply text-only search (MCTS, Predictive Decoding) without dynamically steering attention to task-critical regions during multi-step reasoning.

Method

CoFFT iterates a three-stage loop per reasoning step:

  1. Diverse Sample Generation (DSG): The VLM generates k candidate next-step bundles using temperature sampling (0.4–1.0), each containing up to l foresight reasoning steps.

  2. Dual Foresight Decoding (DFD): Each candidate is scored by a combined metric E = λ·Eatt + (1−λ)·Eprob (λ=0.3). Eatt measures visual coherence via cosine similarity and IoU of top-30% attention regions between the question's relative attention map and the sample's attention map. Eprob measures average log-probability improvement across the l steps relative to the baseline reasoning process. The first step of the top-scoring candidate is appended to the reasoning chain.

  3. Visual Focus Adjustment (VFA): A dual-criteria attention map Acrop is built from (a) question-relevance attention minus already-explored regions (C_rel(V,Q,R_t)) and (b) future-reasoning attention from the selected sample (A_rel(V,s_i)). A sliding window searches regions spanning 40–90% of the original image dimensions and selects the highest-mean-attention crop, subject to an adaptive threshold β that tightens when the two criteria diverge. The crop is upscaled as the next iteration's visual input.

Relative attention normalization (A_rel = Softmax(A(V,Q) / (A(V,D) + ε))) suppresses globally salient but question-irrelevant features throughout.

Key Contributions

  • Training-free iterative visual reasoning loop that couples dynamic image cropping with multi-sample reasoning selection.
  • Dual Foresight Decoding: joint visual-focus score (cosine + IoU of attention maps) and reasoning-progression score (mean log-prob delta across foresight steps).
  • Visual Focus Adjustment with adaptive thresholding that switches between global and local views without requiring extra models or segmentation modules.
  • Demonstrated positive scaling: absolute gains increase from 7B to 32B backbone models.
  • Two new geography-VQA datasets: SeekWorld-China and SeekWorld-Global.

Results

  • CoFFT (Qwen2.5-VL-7B): 48.19% avg vs. 42.72% baseline, 44.68% MCTS, 45.05% Predictive Decoding, 34.91% DyFo, 44.98% ICoT across 7 benchmarks.
  • CoFFT (Qwen2.5-VL-32B): 52.96% avg vs. 47.05% baseline; largest gains on SeekWorld-China (38.61% vs. 24.13%) and MathVision (29.93% vs. 25.33%).
  • CoFFT (InternVL2.5-8B): 45.30% avg vs. 40.84% baseline.
  • CoFFT (LLaVA-Next-7B): 25.54% avg vs. 22.41% baseline.
  • FLOPS (Qwen2.5-VL-7B): CoFFT = 2.38e+14 vs. MCTS = 4.05e+14 (42% cheaper than MCTS) vs. baseline = 8.35e+12.
  • Ablation: removing DFD drops avg by ~2.4%; removing VFA drops avg by ~2.5%, with largest hits on SeekWorld and Charxiv.
  • Still underperforms GPT-4o and Gemini-2.0-Flash on SeekWorld-Global (29.37% vs. ~55–56%).

Limitations

  • Can introduce new errors on problems the base VLM already solves correctly (reasoning interference).
  • Significant compute overhead over greedy baseline (~28× in FLOPS), making real-time deployment costly.
  • Performance on SeekWorld-Global still lags closed-source models, attributed to limited regional geographic knowledge in open-source pretraining data.
  • Hyperparameter sensitivity: performance and cost grow with l and k; optimal values require benchmark-specific tuning.
  • Relies on VLM-internal attention maps, which may be noisy for models with non-standard architectures or sparse-attention variants.

Relevance to Vision-Language Models

CoFFT directly addresses one of the core failure modes of current VLMs—susceptibility to visually salient but task-irrelevant distractors—by coupling the reasoning chain to dynamic, attention-guided image cropping at inference time. It connects to the growing body of test-time compute scaling work (MCTS, Predictive Decoding) but uniquely integrates visual grounding as a first-class signal in the search process rather than treating it post-hoc. The positive scaling trend with model size suggests the method becomes more valuable as base VLMs improve, making it relevant to any researcher studying multimodal reasoning, hallucination reduction, or inference-time augmentation for VLMs.