Skip to content

VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

🕒 Published (v1): 2026-07-14 13:21 UTC · Source: Arxiv · link

Why this paper was selected

Visual token compression for VLMs; training-free LLM-as-encoder approach; addresses core inference efficiency bottleneck

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VisCo is a training-efficient visual token compression framework that repurposes the pretrained VLM itself as an intrinsic compressor via a parameter-sharing asymmetric autoencoder. Learnable memory tokens aggregate visual information during encoding, and layer-wise KV caches are hierarchically injected into the decoder's KV cache, bypassing the need for external compression modules. VisCo achieves 91.8% retained performance on LLaVA-1.5-7B at 32 tokens and remains at 85.3% with a single token.

Problem

Existing visual token compression methods fall into two unsatisfying camps: training-free approaches (FastV, SparseVLM, VisionZip) use heuristic attention or similarity metrics and degrade sharply under high compression ratios; training-based methods with external modules (Q-Former-style, QueCC, Matryoshka Query) force the backbone to adapt to a foreign compression mechanism, requiring costly re-alignment and large-scale instruction tuning. Both paradigms fail to exploit the strong encoding priors already present in pretrained VLMs.

Method

VisCo treats the frozen pretrained VLM backbone \(\Phi(\cdot)\) as both encoder and decoder in an asymmetric autoencoder. The encoding stage augments \(\Phi\) with LoRA adapters \(\theta_\text{LoRA}\) and appends \(N_m \ll N_v\) learnable memory tokens \(X_m \in \mathbb{R}^{N_m \times D}\) after the visual token sequence \(X_v\). Under standard causal masking, each memory token attends to all preceding visual tokens, so the backbone's native attention mechanism performs the aggregation with no custom interaction rules.

At each layer \(l\), the KV entries corresponding to memory tokens are extracted: $\(K_\text{mem}^{(l)} = K^{(l)}[N_v{+}1 : N_v{+}N_m], \quad V_\text{mem}^{(l)} = V^{(l)}[N_v{+}1 : N_v{+}N_m]\)$ and stored in a layer-wise memory bank \(\{K_\text{mem}^{(l)}, V_\text{mem}^{(l)}\}_{l=1}^{L}\), capturing a hierarchy from low-level textures (shallow layers) to abstract semantics (deep layers).

During decoding, the LoRA adapters are removed, and the collected memory KV caches are directly pre-populated into the frozen backbone's KV cache as hierarchical prefixes. The decoder then performs autoregressive generation conditioned solely on \(K_\text{mem}\), \(V_\text{mem}\), and the text prompt \(X_t\): $\(P_d(y \mid K_\text{mem}, V_\text{mem}, X_t;\, \Phi)\)$

Training uses standard teacher-forcing cross-entropy loss on \(\sim\)10% of LLaVA-665K for one epoch, with only \(X_m\) and \(\theta_\text{LoRA}\) (rank 64, \(\alpha=128\)) as trainable parameters.

Key Contributions

  • Intrinsic self-compression: reuses the pretrained VLM as its own compressor with no external module, preserving backbone priors and avoiding costly re-alignment.
  • Hierarchical KV-pass mechanism: transfers layer-wise memory KV caches from encoder to decoder, preserving multi-granularity visual representations rather than collapsing to final-layer outputs only.
  • Asymmetric autoencoder design: LoRA-augmented encoder + frozen decoder with shared backbone weights; only \(X_m\) and \(\theta_\text{LoRA}\) are trained.
  • VisCo+: combining compressed memory tokens with the original visual tokens improves over the base model, demonstrating that memory tokens capture complementary representations.

Results

  • LLaVA-1.5-7B, 32 tokens (from 576): VisCo retains 91.8% Avg, outperforming VisPruner (87.8%), DivPrune (87.2%), PruMerge+ (77.5%), SparseVLM (72.6%), and FastV (57.6%).
  • LLaVA-1.5-7B, 1 token: VisCo retains 85.3% Avg, vs. PruMerge+ at 35.4% and VisPruner at 48.8%; competitive with or exceeding heavily retrained references (VoCo-LLaMA: GQA 57.0 vs. VisCo 58.2; Matryoshka Query: MMB 54.4 vs. VisCo 59.0).
  • Qwen2-VL-2B, 36 tokens (↓66.7% from 144): 95.2% Avg, vs. VisionZip‡ 91.0%, PruMerge+ 80.6%, FastV 84.2%.
  • Qwen2-VL-2B, 18 tokens (↓88.9%): 91.4% Avg, vs. VisionZip‡ 76.1%.
  • Qwen2-VL-7B, 18 tokens: VisCo GQA 61.8 vs. VisionZip‡ 55.3 (+9.1% Avg gap).
  • Ablation (Qwen2-VL-2B, Table 3): Hierarchical KV-Pass outperforms Value-Pass only at all compression ratios; at 18 tokens, GQA jumps from 49.6 (ValuePass) to 59.7 (HierKVPass).
  • VisCo+ (memory + original tokens, Qwen2-VL-2B): surpasses both origin and Direct SFT on MMB (69.6 vs. 67.3), MMB-CN (67.8 vs. 61.8), MME (1491.2 vs. 1465.8), and POPE (84.6 vs. 81.5).

Limitations

  • Evaluated only on models up to 7B parameters; behavior at larger scales (e.g., 70B) is unverified.
  • Training is still required (one epoch on a data subset), so it is not fully plug-and-play like training-free baselines.
  • The encoding pass over the full \(N_v\) visual tokens plus \(N_m\) memory tokens adds a forward-pass overhead at inference time before the compressed decoding; this latency cost is not quantified in the text.
  • MMVet score under VisCo+ drops below origin (37.2 vs. 42.5), indicating the augmented representation does not universally improve all task types.
  • Only evaluated on static-image VLMs; extension to video or interleaved multimodal settings is not addressed.

Relevance to Vision-Language Models

VisCo directly addresses one of the central scalability bottlenecks in modern VLMs—the quadratic cost of self-attention over large visual token sequences—by proposing a principled self-compression approach that exploits the model's own learned representations rather than discarding or averaging tokens heuristically. The hierarchical KV-pass mechanism offers a novel perspective on how the layer-wise structure of transformer LLMs can be repurposed for compression, generalizing across architectures (LLaVA-1.5, Qwen2-VL). The VisCo+ finding—that compressed memory tokens carry information complementary to the original visual tokens—challenges the assumption that compression is purely lossy, with implications for efficient high-resolution and multi-image VLM design. This work sits at the intersection of efficient inference and representation learning in VLMs, relevant to anyone studying token budget management, KV-cache optimization, or multimodal architecture design.