Skip to content

DyMU: Dynamic Merging and Virtual Unmerging for Efficient Variable-Length VLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DyMU is a training-free framework that reduces visual token counts in VLMs dynamically per image based on content complexity rather than resolution alone. It combines Dynamic Token Merging (DToMe) in the ViT encoder with Virtual Token Unmerging (VTU) in the LLM decoder to preserve full-sequence attention dynamics without fine-tuning. Applied to LLaVA-1.5 and LLaVA-OneVision, it achieves 32–85% token reduction with near-lossless performance.

Problem

Current VLMs (including dynamic-resolution models like LLaVA-OneVision, Qwen2.5-VL) generate a fixed number of visual tokens determined solely by image resolution, not content complexity—a 576-token CLIP embedding is identical for a blank image and a dense urban scene. Prior token reduction methods are limited by: (1) fixed compression ratios ignoring image content, (2) requiring full model retraining, or (3) dependence on textual conditioning. None natively produce variable-length visual encoder outputs without training.

Method

DyMU has two components:

Dynamic Token Merging (DToMe): Extends ToMe's bipartite soft matching by replacing fixed per-instance merge counts with a similarity threshold τᵢ per ViT layer. Thresholds are calibrated offline via a single batch-level pass over a diverse image dataset: for each layer i, τᵢ is set to the largest value such that B×rᵢ total edges are merged across the batch. Because ranking is over the full batch, images with more redundant tokens receive more merges. Merged token embeddings are weighted averages tracked via position sets Pᵢ[t]; size-weighted self-attention (log-bias on attention logits) ensures merged tokens are weighted by their absorbed position count.

Virtual Token Unmerging (VTU): Compensates for the LLM's expectation of fixed-length visual sequences by efficiently reconstructing the full N-token attention matrix from the reduced Nᵤₙ unique tokens. It exploits RoPE linearity: the full attention matrix A is expressed in terms of sparse matrices CM, SM (precomputed, ≤N nonzeros each) applied to Qᵤₙ, Kᵤₙ outer products (O(Nᵤₙ²) cost), with cross-terms handled via sparse matmul O(NNᵤₙ). After each self-attention block, outputs at duplicate positions are re-averaged to re-introduce redundancy for subsequent layers. Sequence-independent operators (MLP, LayerNorm) operate directly on the Nᵤₙ unique tokens.

Key Contributions

  • First training-free method that makes visual token length natively variable based on image complexity (not resolution).
  • DToMe: batch-level bipartite threshold calibration enabling content-adaptive token merging within ViT layers.
  • VTU: closed-form reconstruction of full RoPE attention dynamics from a reduced token set, derived analytically; compatible with any RoPE-based LLM.
  • Plug-and-play applicability to ViT+RoPE VLMs (LLaVA-1.5 with CLIP/SigLIP, LLaVA-OneVision AnyRes) without any fine-tuning.
  • Demonstration that DToMe token counts correlate strongly with image complexity (JPEG compression ratio used as complexity proxy).
  • Composability with task-specific tools (background removal, OCR, object detection) for further token budget control.

Results

  • LLaVA-1.5-7B (9 benchmarks: GQA, MMBench, MME, POPE, ScienceQA, SEED-IMG, TextVQA, MMVet, LLaVA-Bench):
  • DyMU-low (89±27 tokens, 84.5% reduction): 97.5% of baseline avg (61.5 vs 63.1)
  • DyMU-mid (195±47 tokens, 66.1% reduction): 99.2% of baseline avg (62.6 vs 63.1)
  • DyMU-high (394±57 tokens, 31.6% reduction): 100.2% of baseline avg (63.2 vs 63.1)
  • Outperforms training-free fixed-length baselines (ToMe, FastV, PDrop, SparseVLM) at comparable token counts.
  • VTU improves performance on 8/9 benchmarks vs. no-VTU variants for both ToMe and DToMe.
  • DyMU-mid achieves 4.3× fewer FLOPs in attention blocks while retaining 99.2% accuracy.
  • LLaVA-1.5 with SigLIP encoder: DyMU-SigLIP-low (90±26 tokens) achieves 96.1% of baseline (59.7 vs 62.1).
  • LLaVA-OneVision (AnyRes, image+video): DyMU-ov-mid at ~25% tokens scores 76.0 MMBench, 70.3 MME vs 79.3/75.8 full baseline; video benchmarks (VidMME, MMBVid) also maintained.
  • Wall-clock speedup is less pronounced than FLOP reduction due to GPU kernel underutilization from decomposed sparse matmuls (acknowledged limitation).

Limitations

  • Performance degrades on information-dense, token-sensitive tasks: TextVQA shows largest drops across all compression levels; OCR and DocVQA are noted as problematic.
  • Wall-clock inference speedup does not match theoretical FLOP reduction—VTU's decomposed sparse operations underutilize optimized GPU kernels (analogous to FlashAttention's kernel engineering challenge).
  • VTU requires RoPE-based LLMs; not applicable to models using other positional encodings.
  • Re-averaging merged positions after each attention block is an approximation that accumulates error across layers.
  • Temporal redundancy reduction in video is unexplored.
  • Complexity-aware selective retention for semantically critical regions (e.g., text) is not addressed.

Relevance to Vision-Language Models

DyMU directly targets one of the central bottlenecks in modern VLMs—the quadratic cost of processing large fixed-length visual token sequences—with a principled, training-free solution that generalizes across architectures. The VTU derivation is a non-trivial theoretical result showing how RoPE attention dynamics can be reconstructed from a compressed token set, which is broadly relevant to any researcher working on efficient VLM inference. Unlike pruning or distillation methods that require retraining, DyMU's plug-and-play nature makes it immediately applicable to closed or open-weight-only VLMs. The dynamic per-image token allocation also opens a new axis of VLM efficiency research: matching compute to content complexity rather than to input dimensions.