Skip to content

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DREAM is a speculative decoding framework for vision-language models that combines cross-attention-based knowledge injection from the target model into a draft model, entropy-guided intermediate feature selection for training supervision, and attention-score-guided visual token compression. Together these achieve up to 3.6× inference speedup over standard autoregressive decoding while outperforming all prior speculative decoding baselines adapted for VLMs.

Problem

Speculative decoding (SD) has been extensively studied for text-only LLMs but is poorly adapted to VLMs, where visual and textual tokens must be fused coherently. Naive application of text-only SD methods (e.g., EAGLE-2, Medusa) to VLMs degrades acceptance rates because existing draft models treat visual features as a flat concatenated sequence, disrupting spatial structure, and lack principled supervision from the rich intermediate representations of the target model. Visual tokens also inflate draft model latency by an average 2.1× versus text-only input.

Method

DREAM trains a small draft model (0.28–0.98B parameters depending on target) alongside a frozen target VLM using three mechanisms:

  1. Cross-attention fusion: At each draft decoding step, last-layer features of the target model (keys/values) are cross-attended to by the draft model's intermediate token embeddings (queries) via learned projection matrices W_Q, W_K, W_V. This preserves multimodal spatial structure instead of flattening visual and textual tokens into a concatenated sequence as EAGLE does. After the first draft token, the draft's own final-layer features are appended to the target feature cache to extend the key-value set.

  2. Entropy-adaptive intermediate feature selection: During training calibration, attention entropy AE(ℓ) = −(1/n²)ΣΣ A_{ℓ,i,j} log A_{ℓ,i,j} is computed for every target layer per sample. The layer with minimum entropy (most focused, highest salience) is selected dynamically as ℓ per step. A smooth-L1 loss distills target features at ℓ into the draft's first decoder block (L_intermed). The full training loss is L_final = λ_feat·L_feat + λ_intermed·L_intermed + λ_KL·L_KL, with weights 0.2, 0.2, 1.0.

  3. Visual token compression (VTC): Importance scores are derived by summing each visual token's attention weights in the target's last layer. The top-scoring fraction (default 75%) of visual tokens is retained for the draft model, reducing its input from q+v to q+⌊v/r⌋ tokens.

Tree-structured draft expansion from EAGLE-2 (width k=4, depth 6, max 32 tokens) is incorporated for parallel candidate generation.

Key Contributions

  • Cross-attention mechanism that injects cached target last-layer features into the draft model, treating visual and textual modalities separately rather than concatenating them.
  • Dynamic per-step selection of the minimum-entropy target layer as distillation supervision, outperforming fixed static-layer baselines and no-intermediate-feature training.
  • Attention-guided visual token subsampling for the draft model that reduces computational overhead while preserving the most salient visual content.
  • Demonstrated up to 3.6× speedup on LLaVA, LLaVA-13B, Pixtral-12B, SmolVLM-2B, and Gemma3-12B across six multimodal benchmarks, consistently surpassing EAGLE-2 by 20–40%.

Results

  • Average speedup at T=0: LLaVA-7B 2.23×, LLaVA-13B 3.06×, Pixtral-12B 2.65×, SmolVLM-2B 2.27×, Gemma3-12B 2.36× vs. standard AR decoding.
  • Peak speedup: 3.68× on LLaVA-13B / MMT-Bench with 75% VTC.
  • Average acceptance length Ï„: 5.51 on LLaVA-7B (T=0), up to 6.40 on individual tasks.
  • vs. EAGLE-2 (best prior VLM-adapted baseline): DREAM improves speedup by ~20–40% across all models/tasks; e.g., ScienceQA LLaVA-7B: DREAM 3.36× vs. EAGLE-2 3.09×; LLaVA-13B: DREAM 3.20× vs. EAGLE-2 2.64×.
  • Ablation — cross-attention removal: Dropping the cross-attention block reduces speedup to 0.41–0.48× of DREAM baseline, the largest single-component drop.
  • Ablation — VTC at 75% retention: Gains 7% additional speedup with <2% reduction in Ï„; dropping to 50% causes 10–25% Ï„ degradation.
  • Ablation — dynamic vs. static layer selection: Dynamic entropy outperforms best static baseline (S-75%, layer 30/40) on all three datasets.
  • Tree vs. chain decoding: Tree structure adds an average 1.32× speedup on top of DREAM's other components.

Limitations

  • Evaluated only on NVIDIA A100 GPUs; performance on other hardware (edge devices, CPUs, other GPU families) is unknown.
  • Gemma3-12B achieves notably lower acceptance length (Ï„ = 2.36) due to its more complex cross-modal processing pathway, suggesting DREAM's gains are architecture-dependent.
  • OCR-intensive and fine-grained character recognition tasks (MathVista, OCRBench) yield lower Ï„ and speedup across all methods, including DREAM; character-level drafting remains an open challenge.
  • Training requires offline calibration (entropy computation over all samples) before draft model training, adding a data preprocessing stage.
  • Evaluation limited to six benchmarks and five VLMs; broader dataset and architecture coverage is left to future work.

Relevance to Vision-Language Models

DREAM directly addresses the inference efficiency bottleneck that constrains VLM deployment, a practical barrier as VLMs scale into the 7B–13B parameter range. The cross-attention fusion mechanism highlights a fundamental design insight: multimodal speculative decoding requires modality-aware feature injection rather than the flat token concatenation sufficient for text-only SD, a finding with implications for how future multimodal draft models should be architected. The entropy-based intermediate layer selection connects to a growing body of work showing that mid-layer representations in transformers are often more task-informative than final-layer outputs, with applicability to VLM distillation more broadly. Results spanning LLaVA, Pixtral, SmolVLM, and Gemma3 suggest the framework transfers across diverse VLM backbone designs.