Skip to content

Self-Prophetic Decoding to Unlock Visual Search in LVLMs

🕒 Published (v1): 2026-05-27 17:01 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; self-prophetic decoding enables visual search in LVLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SeProD (Self-Prophetic Decoding) is a training-free, plug-and-play inference framework that pairs a post-training visual-search LVLM (the "search model") with its pre-training base counterpart (the "prophet model") to address two failure modes in multi-step visual search: degraded intrinsic capabilities and long-context interference. The prophet generates single-step capability-specific prefixes that the search model selectively absorbs via probability-based sampling, preserving coherent multi-turn reasoning without added computational overhead.

Problem

Visual-search post-training of LVLMs introduces two compounding problems. First, joint optimization for task completion causes catastrophic degradation of individual intrinsic capabilities: grounding drops by 49.3%, OCR by 2.3%, spatial understanding by 10.9%, and counting by 3.0% relative to the pre-training base. Second, long multi-step trajectories accumulate interference—erroneous intermediate steps and irrelevant context propagate forward; manually removing irrelevant turns from trajectories recovers 5.56%, 2.24%, and 5.66% on the three splits of VisualProbe-test, demonstrating the severity of context pollution.

Method

SeProD couples two models: the post-training LVLM (search model) that drives global multi-turn reasoning and grounding/answering mode decisions, and its pre-training base (prophet model) that produces single-step, capability-specific prophetic prefixes.

Search-to-prophet: At each reasoning turn \(i\), the search model passes the current cropped image \(I_i\) and a task-specific query \(Q^p\) (grounding-verification query \(Q^g\) or original query \(Q\)) to the prophet, which responds in one forward pass: $\(p_p(O_i | I_i, Q^p) = \prod_{j=1}^{L_d} p_p(o_{i,j} | I_i, Q^p, o_{i,<j})\)$

Prophet-to-search probabilistic decoding: Inspired by LLM speculative decoding, the search model evaluates all prophet tokens \(O_i\) in parallel and accepts token \(x_j\) from the prophet if the acceptance score \(s_j \geq \tau\) (a threshold derived from both models' adjusted distributions); otherwise it falls back to sampling from the search model: $\(x_j \sim \begin{cases} p_p(x_j | I_i, Q^p, x_{<j}), & \text{if } j < \min\{j \mid s_j < \tau\} \\ p_s(x_j | H_i, x_{<j}), & \text{otherwise} \end{cases}\)$

Grounding mode prefixes steer next-turn reasoning \(R_{i+1}\); answering mode prefixes generate the final answer \(A_i\) on-the-fly. Prefix evaluation is fully parallelizable, so a smaller prophet model incurs minimal overhead.

Key Contributions

  • Identifies self-regulation between pre- and post-training LVLMs as a principle for recovering degraded intrinsic capabilities and reducing long-context interference without retraining.
  • First application of probability-based prophetic sampling (adapted from speculative decoding) as an interface between pre- and post-training LVLMs for LVLM reasoning and visual search.
  • SeProD framework that is training-free, plug-and-play, and compatible with any intrinsic-extended LVLM (Pixel Reasoner, DeepEyes, Mini-o3 demonstrated).
  • Achieves state-of-the-art across all 12 splits of 4 visual search benchmarks (V*Bench, HR-Bench, VisualProbe-test, MME-RealWorld) and improves general VQA benchmarks, with no added computational overhead due to parallel prefix acceptance.

Results

  • Consistent improvement over three post-training LVLM baselines (Pixel Reasoner, DeepEyes, Mini-o3) across all 12 evaluation splits of 4 benchmarks.
  • Removing irrelevant context from trajectories (motivation experiment) recovers +5.56%, +2.24%, +5.66% on VisualProbe-test easy/medium/hard splits, validating the long-context interference hypothesis.
  • Gains also on ScienceQA, OCRBench, CVBench, and MME-RealWorld (specific numbers not fully reported in the provided text excerpt).
  • Parallel prophetic acceptance introduces no measurable computational overhead; prophet can be a smaller model to improve efficiency further.

Limitations

  • Results on general VQA benchmarks and full numeric comparisons with baselines are in truncated sections of the paper, limiting external verifiability from the provided text.
  • The threshold \(\tau\) is a manual hyperparameter; sensitivity analysis not described in the available text.
  • Requires access to the base pre-training checkpoint of each target post-training model, which may not always be publicly available.
  • Evaluated on a specific class of intrinsic-extension LVLMs; applicability to tool-augmentation paradigms is not claimed.

Relevance to Vision-Language Models

SeProD directly addresses a known failure mode in RLVR/RLHF fine-tuning of VLMs: catastrophic forgetting and capability interference, which are active concerns as the community scales visual reasoning post-training. The probability-based prophetic sampling mechanism is a novel adaptation of speculative decoding—a technique from pure LLM inference—to cross-model distribution alignment in multimodal settings, opening a new design axis for inference-time VLM enhancement. It connects to the broader "thinking-with-images" line of work (DeepEyes, Mini-o3, Pixel Reasoner) while being uniquely orthogonal to those training-based methods, functioning as a drop-in inference wrapper. For VLM researchers, the quantified single-step capability degradation data (−49.3% grounding) provides concrete diagnostic evidence relevant to any post-training recipe.