Skip to content

ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ViSpec is a speculative decoding framework tailored for VLMs that achieves the first substantial inference speedup in this setting (up to 3.22×). It addresses the core failure mode of prior methods—small draft models drowning in redundant image tokens—via compressed visual embeddings and a persistent global visual feature injection. Experiments across four VLMs and eight benchmarks consistently outperform adapted text-only baselines (Medusa, EAGLE-2).

Problem

Speculative decoding is well-established for LLMs (3–4× speedup) but prior applications to VLMs achieved only ≤1.5×. The gap arises because shallow draft models cannot filter the high redundancy in image token sequences: a single-layer Transformer's attention to informative tokens vanishes as redundant tokens dominate (αᵢᵤ → 0 as R → ∞), and shallow networks lack the depth to handle nested sequence complexity. Additionally, multimodal datasets with long assistant responses—required to train effective draft models—are scarce.

Method

ViSpec introduces three complementary mechanisms built on top of the EAGLE-2 draft-verify framework:

  1. Image embedding compression (vision adaptor): A lightweight Q-Former-style Transformer encoder with fixed learnable query vectors compresses hundreds/thousands of raw image embeddings into as few as 1 compact token. These compressed tokens are prepended to the draft model's attention input, preserving relative spatial positions while eliminating redundancy.

  2. Global visual feature injection: A single global feature vector g is derived from the vision adaptor's final output and added (via a learned projection Wg) to the hidden state of every subsequent text token in the draft model (ftaug = ft + Wg·g). This counters the "lost-in-the-middle" degradation in long generations by giving the draft model persistent image context regardless of sequence length.

  3. Synthetic long-response dataset generation: Existing VQA and captioning datasets are repurposed by modifying prompts to elicit verbose responses from the target VLM. Draft model training uses sampling (not greedy) plus multi-token prediction (à la DeepSeek) to prevent shortcut learning from target hidden states, mimicking EAGLE's noisy augmentation effect.

The draft model's own hidden states f′ are also fed back as input, enabling self-correction without manually curated supervision.

Key Contributions

  • First speculative decoding framework to achieve meaningful VLM acceleration (claimed first >1.5× speedup).
  • Theoretical analysis showing why single-layer draft models fail on redundant image sequences, motivating compression.
  • Dual vision integration: attention-level compressed token injection + token-level global feature augmentation.
  • Synthetic dataset generation strategy with sampling + multi-token prediction to avoid shortcut learning.
  • Empirical validation across LLaVA-v1.6-7B/13B and Qwen2.5-VL-3B/7B on 8 diverse benchmarks.

Results

  • LLaVA-v1.6-7B, temp=0: ViSpec avg 2.58× vs. EAGLE-2 1.62× vs. Medusa 1.42×; peak 3.22× on COCO Captions.
  • LLaVA-v1.6-13B, temp=0: ViSpec avg 2.38×; peak 2.82× (COCO Caps).
  • Qwen2.5-VL-3B, temp=0: ViSpec avg 1.87× vs. EAGLE-2 1.39× vs. Medusa 1.14×.
  • Qwen2.5-VL-7B, temp=0: ViSpec avg 1.80× vs. EAGLE-2 1.40× vs. Medusa 1.11×.
  • At temp=1, ViSpec maintains advantage: LLaVA-7B avg 2.14× vs. EAGLE-2 1.54×.
  • Ablation (LLaVA-7B, COCO Caps): image compression alone +30% speedup over baseline; global injection adds +7%; dataset generation adds another +30%.
  • Optimal compressed embedding count is 1; increasing to 64 adds computation without acceptance-length gain.
  • Longer output sequences yield higher speedup (COCO Caps 236 tokens → 3.21×; GQA 46 tokens → 2.22×).

Limitations

  • Absolute speedup still trails text-only speculative decoding (EAGLE achieves 3–4× on LLM-only tasks).
  • Speedup degrades at temperature=1 and on models with large vocabularies (Qwen2.5-VL) due to harder token prediction.
  • Training requires 8 GPUs; synthetic data quality is bounded by target model's output diversity.
  • Vision adaptor hyperparameters (query count, architecture) may need tuning per VLM family.
  • No evaluation on video or multi-image settings despite VLMs supporting them.

Relevance to Vision-Language Models

ViSpec directly addresses inference efficiency—a critical deployment bottleneck as VLMs scale in parameter count and image resolution. The theoretical insight that image redundancy systematically degrades shallow draft models provides a principled explanation for why text-only speculative decoding recipes fail to transfer to VLMs. The Q-Former-inspired compression module is architecturally compatible with any LLaVA-style VLM, making the framework broadly applicable to the dominant VLM design pattern. For the VLM community, this opens a complementary acceleration axis (alongside token pruning and vision encoder compression) that preserves lossless generation fidelity.