Skip to content

TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding

🕒 Published (v1): 2026-07-13 06:06 UTC · Source: Arxiv · link

Why this paper was selected

Multimodal speculative decoding with gated routing; practical inference speedup for VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TIGER is a multimodal speculative decoding framework that addresses why standard speculative decoding underperforms in VLMs: drafters diverge on vision-critical tokens, and training optimizes imitation rather than the actual speedup-governing metric. It introduces text-conditioned dynamic visual token routing to give the drafter contextually relevant evidence, and trains with a verifier-acceptance-length reward via GRPO-style policy optimization.

Problem

Speculative decoding—where a lightweight drafter proposes tokens verified by a large target model—yields limited speedup in VLMs. Two coupled mismatches cause this: (1) the drafter receives the full or fixed-compressed visual token set, which may include irrelevant or noisy evidence for the current generation context, causing early divergence on vision-critical tokens (OCR strings, chart values, object counts, grounded entities); (2) existing training objectives minimize imitation loss, not the quantity that directly governs speedup—the verifier-accepted prefix length \(\bar{A} = \mathbb{E}[A_i]\), which determines the expected number of expensive verifier rounds via \(\mathbb{E}[M] \approx N / \bar{A}\).

Method

TIGER operates in two stages per speculative round:

Text-conditioned Visual Gated Routing. Given the drafter's hidden state \(h_j\) at the end of the current textual prefix, a learnable projection produces a routing query \(q_j = W_q h_j\). Each visual token \(\bar{v}_i\) in the cached visual bank (including an optional pooled global token) is scored by \(s_i = q_j^\top W_v \bar{v}_i / \sqrt{d}\), and the top-\(k\) tokens are selected as the routed visual context \(V_j^{\text{route}}\). The drafter then generates the speculative block conditioned on \(X_j = [E_t(T_{1:j}); V_j^{\text{route}}]\). The verifier always operates on the full visual context, so routing affects only the drafter interface.

Acceptance-Aligned Training. Training proceeds in two stages: (1) KD warm-start to obtain a stable reference policy \(q_{\text{ref}}\); (2) for each training example, \(K\) candidate draft blocks \(\hat{Y}^{(1)}, \ldots, \hat{Y}^{(K)}\) are sampled and evaluated against the verifier under the standard acceptance test. The reward for each candidate is its verifier-accepted prefix length \(R^{(k)} = A^{(k)} = \sum_{u=1}^{\gamma} \prod_{m=1}^{u} a_m^{(k)}\). Rewards are normalized within the group, and a GRPO-style objective is optimized: $\(\mathcal{L}_{\text{accept}} = -\sum_{k=1}^{K} \hat{g}^{(k)} \log q_\theta(\hat{Y}^{(k)} | X_j) + \beta \, \mathrm{KL}(q_\theta \| q_{\text{ref}})\)$ KL anchoring prevents policy drift from the distillation initialization.

Key Contributions

  • Text-conditioned visual gated routing that dynamically selects a sparse, context-relevant subset of visual tokens for each speculative block rather than using a fixed or full visual interface.
  • An acceptance-aligned training objective that uses verifier-accepted prefix length as a reward signal, trained via group-based relative policy optimization (GRPO-style) with KD warm-start and KL anchoring.
  • Experimental protocol that cleanly separates exact verifier-side speculative decoding evaluation (AccLen, Speedup) from auxiliary quality–latency analyses of different drafter-side visual interfaces.

Results

  • LLaVA-v1.6-7B, COCO Caption: TIGER AccLen = 3.37, Spd = 3.24× vs. next-best ViSpec (3.34, 3.20×) and EAGLE-3 (3.30, 3.18×).
  • LLaVA-v1.6-7B, TextVQA: TIGER AccLen = 2.98, Spd = 2.91× vs. EAGLE-3 (2.97, 2.78×) and RACER (2.90, 2.80×).
  • LLaVA-v1.6-7B, MMBench: TIGER AccLen = 2.67, Spd = 2.57× vs. TokenRecycling (2.54, 2.43×).
  • LLaVA-v1.6-13B, COCO Caption: TIGER AccLen = 3.25, Spd = 2.19× vs. EAGLE-3 (3.24, 2.15×).
  • Qwen3-VL-4B and 8B: TIGER achieves tied-best AccLen and speedup across benchmarks.
  • Auxiliary quality–latency (COCO Caption): TIGER routing achieves Caption Acc = 0.75, Norm. Lat. = 0.62 on LLaVA-v1.6-7B vs. VisPruner (0.69, 0.76) and SparseVLM (0.56, 0.82).
  • Routing sparsity ablation (LLaVA-v1.6-7B): \(k=8\) is optimal (AccLen = 3.37, Spd = 3.24×, Caption Acc = 0.75); \(k=32\) and full visual tokens degrade all metrics.
  • Component ablations (LLaVA-v1.6-7B, COCO Caption): Removing visual gated routing reduces AccLen to 2.13; removing acceptance alignment to 2.07; removing KD warm-start on Qwen3-VL-4B collapses speedup to 1.03×.

Limitations

  • Speedup gains over strong speculative baselines (EAGLE-3, RACER, TokenRecycling) are "consistent but generally modest," suggesting diminishing returns as draft quality saturates.
  • Long-video decoding remains challenging; TIGER shows only moderate improvements in that regime.
  • TIGER requires training (KD warm-start + policy optimization), unlike plug-and-play methods such as RACER and TokenRecycling.
  • Routing adds per-round overhead (query projection + scoring + top-k selection), though the paper reports it is included in end-to-end latency measurements.
  • Evaluation is limited to LLaVA and Qwen-VL model families; generalization to other VLM architectures is not demonstrated.

Relevance to Vision-Language Models

TIGER directly targets a VLM-specific inference bottleneck: the mismatch between static visual token compression and the context-varying nature of vision-critical generation. By framing visual token selection as a dynamic, text-conditioned routing problem within the speculative decoding loop, it offers a practical path to reducing VLM serving costs without changing the verifier or degrading output quality. The acceptance-aligned training strategy—using verifier acceptance as a reward signal via GRPO—is a novel application of RL-from-verifier-feedback to VLM inference efficiency, complementing the growing literature on preference-aligned VLM training. For researchers tracking VLMs, the work also highlights that OCR, counting, and visually-grounded reasoning tasks impose qualitatively different drafting difficulties, which may inform future work on task-adaptive inference.