Skip to content

Spectral Heat Flow for Conservative Token Condensation in Vision-Language Models

🕒 Published (v1): 2026-07-12 08:12 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; spectral-heat-flow token condensation; directly cuts VLM inference cost without info loss

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpecFlow is a training-free visual token condensation framework for VLMs that replaces destructive Top-K pruning with spectral heat diffusion on a kNN token graph, adaptive quadtree spatial partitioning, and coreset sink aggregation. It retains 95.6% of LLaVA-1.5's original performance while pruning 88.9% of visual tokens (576→64). The method is plug-and-play and FlashAttention-compatible.

Problem

Existing attention-score-based Top-K token pruning for VLMs fails under high compression due to two structural failure modes: (1) spatial fragmentation — spiky [CLS] attention produces holes within objects because token importance is scored independently rather than coherently; (2) contextual collapse — saliency-driven pruning over-preserves foreground while aggressively discarding background context essential for spatial reasoning. Neither spatial coherence nor statistical conservation of pruned information is enforced.

Method

SpecFlow operates in three stages, all training-free:

1. HeatFlow (CLS-seeded spectral diffusion). Visual tokens are nodes in a sparse kNN graph whose edge weights are softmax-normalized cosine similarities (not raw patch self-attention, which is avoided due to anomaly-token artifacts in CLIP-ViTs). The [CLS] outgoing attention at a chosen layer seeds an energy vector \(e^{(0)}\). Restart diffusion iterates: $\(e^{(t+1)} = (1-\alpha)e^{(0)} + \alpha W^\top e^{(t)}\)$ converging to \(e^\star\) that minimizes a Dirichlet-regularized objective balancing seed fidelity and graph smoothness (Proposition 3.1). This smooths spiky attention into a region-coherent importance field at \(O(Nk)\) per iteration.

2. Quadtree allocation. The 1D energy vector is reshaped into a 2D map \(E \in \mathbb{R}^{H \times W}\). Adaptive quadtree partitioning recursively splits crops where \(\sigma(c) > \delta\) (high energy variance), yielding finer tiles in salient regions and coarser tiles in homogeneous background. Per-leaf token budgets \(q_c\) are set energy-proportionally: $\(\hat{q}_c = K \cdot \frac{M(c)}{\sum_{c'} M(c')}\)$ which is proven to be the unique optimizer of a concave proportional-fairness objective (Proposition 4.1). Within each leaf, the top-\(q_c\) energy tokens are retained.

3. Coreset sink tokens. Pruned tokens \(Z_p = Z[\bar{S}]\) are compressed into two sink vectors appended to the kept sequence: a mean sink \(u_\text{mean} = \frac{1}{|\bar{S}|}\sum_{i\in\bar{S}} Z_i\) and a residual sink \(u_\text{res} = Z_{\arg\max_{i\in\bar{S}} \|Z_i - u_\text{mean}\|_2}\) (the most diverse outlier). Final input to the LLM decoder: \(Z' = [\tilde{Z}; u_\text{mean}; u_\text{res}]\).

Key Contributions

  • Formal diagnosis of two failure modes of attention-based Top-K pruning: spatial fragmentation and contextual loss.
  • Spectral heat diffusion on a feature-similarity kNN graph (not raw attention) as a principled, training-free importance propagation mechanism; convergence characterized as a Dirichlet-regularized convex problem.
  • Adaptive quadtree partitioning with energy-proportional budget allocation (proved to solve a proportional-fairness program), enforcing spatial coverage guarantees.
  • Conservative condensation via two coreset sink tokens that statistically summarize discarded tokens with negligible overhead.
  • End-to-end plug-and-play design: no fine-tuning, FlashAttention-compatible, one-time pruning overhead amortized across layers.

Results

All numbers on LLaVA-1.5 7B against baselines ToMe, FastV, LLaVA-PruMerge, PDrop, HiRED, VisionZip, SparseVLM, DART, HoloV, across GQA, MMBench, MMB-CN, MME, POPE, SQA, VQAv2, TextVQA, VizWiz:

  • 576→192 tokens (66.7% pruned): SpecFlow 98.7% relative performance — best among all methods; next best HoloV 98.2%, SparseVLM 98.5%.
  • 576→128 tokens (77.8% pruned): SpecFlow 97.8% — best; next best SparseVLM 97.5%, DART 96.8%.
  • 576→64 tokens (88.9% pruned): SpecFlow 95.6% — best; next best DART 94.9%, HoloV 94.5%. At this extreme ratio SpecFlow surpasses HiRED (94.5%) and SparseVLM (93.9%).
  • LLaVA-NeXT (up to 2880→320 tokens, 88.9% pruned): SpecFlow retains 95.8% of unpruned performance, best among compared methods.
  • Video QA (VideoLLaVA, 2048 tokens): SpecFlow achieves the best average on MSVD-QA and MSRVTT-QA (full numbers truncated in provided text).
  • On MMBench and SQA at 66.7% compression, SpecFlow surpasses the unpruned baseline.
  • Prefill FLOPs saving at pruning ratio \(R\): \(\Delta_\text{pre} \approx 2R - R^2\) (superlinear); decode saving \(\Delta_\text{dec} \approx R\).

Limitations

  • kNN graph construction has worst-case \(O(N^2 h)\) FLOPs; for very large token counts (e.g., high-resolution or multi-frame video) this may become a bottleneck despite amortization across layers.
  • The quadtree splitting threshold \(\delta\) and minimum crop size \(m\) are hyperparameters requiring tuning per architecture/resolution.
  • Evaluated primarily on LLaVA-1.5 and LLaVA-NeXT; generalization to decoder-only or non-CLIP vision encoders is not fully demonstrated in the provided text.
  • Sink token approach compresses all discarded tokens into just two vectors; diversity preservation is limited to a single residual token, which may be insufficient at very high token counts.
  • No training-time optimization means the method cannot recover from quality loss in cases where structural priors of the kNN graph diverge from task-relevant semantics.

Relevance to Vision-Language Models

SpecFlow directly targets the inference efficiency bottleneck of VLMs arising from quadratic attention cost over dense visual token sequences — a problem that grows worse as models adopt higher-resolution encoders (LLaVA-NeXT: 2,880 tokens) and video inputs. By grounding token reduction in spectral graph theory rather than heuristic Top-K, the work advances the theoretical understanding of when and why pruning preserves multimodal reasoning. The coreset sink mechanism is relevant to the broader line of work on token merging (ToMe) and representation compression, offering a statistically grounded alternative that maintains distributional diversity. For VLM researchers, SpecFlow demonstrates that spatial structure and contextual coverage are first-class constraints that any competitive token reduction scheme must explicitly enforce.