Compress & Cache: Vision token compression for efficient generation and retrieval¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

C&C (Compress & Cache) is a visual token compression method for large vision-language models (LVLMs) that offloads compression to an offline indexing stage rather than performing it on-the-fly during inference. It uses a "double-forward pass" strategy where the LVLM's own LLM compresses 576 vision tokens into a small set of summary tokens, trained jointly with autoregressive and contrastive losses to support both generation and retrieval. This achieves 2× better compression rates than prior art for generative tasks while simultaneously setting new state-of-the-art on image retrieval benchmarks.

Problem¶

Prior LVLM token compression methods operate on-the-fly during inference, limiting compressor capacity and making them incompatible with retrieval-augmented generation (RAG) pipelines where images are available a priori. Furthermore, no existing compression approach produces representations suitable for both generative tasks (VQA, captioning) and discriminative tasks (image-text retrieval), forcing RAG systems to maintain separate retrieval and generation models.

Method¶

C&C decouples compression from inference via two stages:

Offline compression (first forward pass): Learnable summary tokens \(H_r \in \mathbb{R}^{k' \times d}\) (with \(k' \ll 576\)) are concatenated with image tokens \(H_v\) and a fixed prompt ("Summarize the image in a few words.") and passed through the LLM. The LLM acts as a bottleneck, transforming the summary tokens into compressed visual representations \(H_{cv}\).

Online generation (second forward pass): The same LLM (shared weights) takes only \(H_{cv}\) and the language instruction as input, trained with next-token prediction loss \(L_{AR}\) to reconstruct the full information flow.

A contrastive loss \(L_{disc}\) (CLIP-style InfoNCE) is applied over \(H_{cv}\) after the first pass using CC3M caption pairs, with text embeddings computed analogously. The total loss is \(L_{total} = L_{AR} + L_{disc}\).

Stage-specific LoRA adapters (rank 64) are used for the compression and generation passes separately, since the optimal weight adjustments differ between stages (confirmed by Frobenius norm analysis). Training runs for 10,000 iterations on 24 AMD MI300X GPUs with batch size 1024 and AdamW (lr=2e-4, cosine decay), sampling generative (LLaVA-665K) and discriminative (CC3M) data at 1:1 ratio.

Key Contributions¶

Double-forward pass training strategy that leverages the LVLM's own LLM for self-compression of visual tokens into cached summary tokens
Unified compressed representation supporting both autoregressive generation and CLIP-style discriminative retrieval via joint \(L_{AR} + L_{disc}\) training
Stage-specific LoRA adapters that allow distinct weight updates for the compression vs. generation forward passes
Offline caching paradigm that decouples compression cost from inference, enabling single-model visual RAG with no separate retriever

Results¶

Generative benchmarks (vs. 576-token LLaVA-1.5 baseline, at 32 tokens): - C&C at 32 tokens matches LLaVA-1.5 on POPE (85.9 vs. 85.9) and nearly matches on GQA (61.6 vs. 62.0) and MMB (64.6 vs. 64.3) - C&C at 16 tokens outperforms all prior methods at 36 tokens (e.g., GQA 61.0 vs. QueCC 59.0 at same 16 tokens); achieves 2.25× fewer tokens than competitors at equivalent accuracy - C&C at 4 tokens: GQA 58.6 vs. QueCC 56.5 and TokenPacker 56.2

Captioning (CIDEr, 32 tokens): C&C scores 78.9/113.1/105.9 on Flickr30K/COCO/NoCaps vs. Matryoshka Query 69.5/101.3/90.0; at 4 tokens COCO CIDEr 111.4 vs. Matryoshka Query 81.0

Discriminative/retrieval (zero-shot Flickr30K R@1, image retrieval): C&C 83.8 vs. VLM2Vec 80.1, E5-V 76.7, LLaVA-1.5 59.6; trained on ~3M samples vs. EVA-CLIP's 2,700M

SugarCrepe compositionality: C&C (Replace/Swap/Add): 81.9/88.0/89.3 vs. EVA-CLIP 85.9/70.4/86.7; C&C leads on Swap and Add

Visual RAG (MRR@10 average): C&C 0.89B achieves 80.38 vs. VisRAG-Ret 3.43B at 77.91; uses 24× fewer vision tokens for generation while nearly matching uncompressed LLaVA-OV-0.5B accuracy

Ablation — loss functions (16 tokens, MME): Discriminative-only 624.3; Generative-only 1420.1; Both 1470.0

Ablation — double vs. single forward (MMB): Double-fwd 64.4 vs. single-fwd 61.3

Limitations¶

Requires an upfront offline indexing stage; unsuitable for scenarios demanding real-time, on-the-fly compression without preprocessing
Compressor is more expensive than on-the-fly methods during the indexing phase (roughly equivalent FLOPs to a full LLaVA forward pass), though this cost is amortized across queries
Inherits biases from the pre-trained LVLM (LLaVA-1.5/LLaVA-OneVision) used as the backbone; no mitigation proposed
Evaluated primarily on LLaVA-1.5 (7B) and LLaVA-OneVision (0.5B); generalization to other LVLM architectures not demonstrated

Relevance to Vision-Language Models¶

C&C directly addresses one of the core efficiency bottlenecks in VLMs—the quadratic cost of processing hundreds of visual tokens in the LLM—by reframing compression as an offline summarization task performed by the model itself. The key insight that the LLM's input and output spaces are sufficiently aligned to enable self-compression via a bottleneck forward pass is broadly applicable and could be adopted in any autoregressive LVLM architecture. The dual-loss training scheme (autoregressive + contrastive) is notable for unifying generative and discriminative VLM capabilities in a single compact representation, addressing the long-standing trade-off between models trained for generation vs. retrieval. For researchers tracking VLMs, C&C establishes a new paradigm for efficient multimodal RAG—single-model retrieval-then-generation—with implications for on-device deployment and document understanding systems.