Skip to content

EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling

🕒 Published (v1): 2026-04-18 17:52 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Evolutionary semantic labeling for visual token compression; CVPR 2026 efficiency

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EvoComp is a learned visual token compression framework for MLLMs that trains a lightweight encoder-only transformer compressor using supervision labels generated by an evolutionary search algorithm. It selects task-relevant, semantically diverse visual tokens by minimizing the MLLM's output loss directly, achieving 99.3% accuracy retention under 3× compression with up to 1.6× inference speedup on mobile hardware.

Problem

Existing visual token compression methods rely on proxy heuristics—attention scores suffer from position bias and FlashAttention incompatibility; similarity-based filtering removes redundancy but ignores semantic importance; text-conditioned methods still depend on attention heuristics rather than optimizing directly for output quality. No prior method provides principled, task-aligned supervision labels for training a dedicated compressor module.

Method

EvoComp inserts a lightweight, single-layer encoder-only transformer compressor between the MLLM's alignment module and the LLM. Training proceeds in two stages:

Label construction via evolutionary labeling. For each training sample with aligned visual tokens \(V = \{v_i\}_{i=1}^n\) and text tokens \(T\), an evolutionary algorithm searches for a binary mask \(m \in \{0,1\}^n\) minimizing the LLM's task loss \(\mathcal{L}(m)\) over the retained tokens. To enforce semantic diversity, a semantic grouping strategy partitions tokens into subsets by finding each token's nearest LLM vocabulary embedding \(e_j\) (via cosine similarity \(S_{ij} = \frac{v_i \cdot e_j}{\|v_i\|_2\|e_j\|_2}\)), grouping tokens sharing the same nearest anchor, and permitting only one selection per group. The evolutionary search (population size \(q=48\), parents \(p=12\), iterations \(L=10\)) uses crossover (probability 0.9, half-and-half sub-mask recombination) and mutation (probability 0.2 per sub-mask, shifting the selected position left/right by one).

Compressor training. The compressor \(f_\phi\) (bidirectional transformer) + classifier \(g_\psi\) (linear) is trained with a combined loss: $\(\mathcal{L}(\varphi,\psi) = \mathcal{L}_{\text{GHM-C}}(\varphi,\psi) + \alpha\,\mathcal{L}_{\text{CS}}(\varphi)\)$ - GHM loss reweights token gradients by their density \(GD(g_i)\) to address class imbalance (few positive/retained tokens) and difficulty imbalance (easy negatives dominate). - Cosine similarity loss \(\mathcal{L}_{\text{CS}}\) penalizes representational similarity between retained and discarded tokens, sharpening the classifier's decision boundary.

At inference, the top-\(r\) tokens by retention probability are selected in a single forward pass, with pruning applicable at any LLM layer.

Key Contributions

  • Evolutionary labeling framework that generates token-level supervision by directly minimizing MLLM task loss, without backpropagation through the LLM.
  • Vocabulary-anchored semantic grouping that enforces non-redundancy in the search space and reduces its combinatorial size.
  • GHM + cosine similarity loss combination tailored to the severe class and difficulty imbalance inherent in token classification.
  • Plug-and-play compressor requiring no modification to the visual encoder, alignment module, or LLM; supports flexible early/late-layer pruning.
  • Cross-model transferability: compressor trained on LLaVA-1.5-7B transfers to LLaVA-1.5-13B and Qwen2.5-VL-7B.

Results

All results use LLaVA-1.5-7B (576 base tokens) unless noted.

  • 3× compression (192 tokens): EvoComp (l=2) achieves 99.3% average accuracy vs. baseline; best competitor VisionZip reaches 98.8%.
  • 4.5× compression (128 tokens): EvoComp (l=0) 97.8%, EvoComp (l=2) 98.0%; DART (best baseline) 97.5%.
  • 9× compression (64 tokens): EvoComp (l=0) 94.9%, EvoComp (l=2) 93.9%; DART 92.8%, VisionZip 94.2%.
  • High-resolution (LLaVA-NeXT-7B, 94.4% compression, 160/2880 tokens): EvoComp (l=0) 92.1% vs. DART 89.6%, SparseVLM 89.7%.
  • Cross-model transfer (LLaVA-1.5-7B → 13B, 64 tokens): EvoComp (l=2) 94.4% vs. DART 94.1% (computed on 13B directly).
  • Mobile speedup: 1.6× at 3× compression, 2.0× at 9× compression on smartphone NPU.

Limitations

  • Evolutionary label construction is computationally expensive at training time (requires repeated parallel LLM inference per training sample across \(q \times L\) mask evaluations).
  • The compressor is trained per-MLLM; a new compressor must be trained (or transferred) for each target model.
  • Evaluation is limited to six standard VQA-style benchmarks; performance on fine-grained spatial reasoning, OCR-heavy, or long-context tasks is not reported.
  • Transfer to architectures with fundamentally different token counts or visual encoders (e.g., interleaved video MLLMs) is not demonstrated.

Relevance to Vision-Language Models

EvoComp directly addresses the inference bottleneck that limits deployment of high-resolution and multi-image MLLMs on edge devices—a key practical constraint for VLM productionization. Unlike prior work that repurposes internal attention patterns as compression signals, EvoComp establishes a principled training paradigm where token selection is supervised by task loss itself, moving the field toward learned, output-aligned compression. The vocabulary-anchored grouping strategy is a novel use of the LLM's own embedding space for structural guidance, complementing emerging ideas around cross-modal token alignment. The demonstrated transferability across model sizes and architectures suggests that the learned compression policy captures general visual redundancy patterns rather than model-specific artifacts.