Skip to content

Towards General Continuous Memory for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoMEM proposes using a VLM as its own continuous memory encoder, compressing retrieved multimodal/multilingual knowledge into just 8 dense embeddings via a lightweight Q-Former rather than concatenating raw image-text tokens. Fine-tuning only 1.2% of parameters on 15.6K self-synthesized samples, the frozen inference-time VLM gains a plug-and-play memory module. This yields average gains of +8.0% and +7.7% on six English multimodal reasoning benchmarks for Qwen2-VL and Qwen2.5-VL respectively.

Problem

Standard RAG for VLMs concatenates retrieved image-text pairs as discrete tokens, which can balloon context length to tens of thousands of tokens per query (up to 11,427 tokens per image in Qwen2.5-VL), causing performance degradation rather than improvement. Token pruning mitigates length but discards content, hurting comprehension. Existing continuous-memory encoders require large training corpora (500K–6.82M samples) and extensive parameter updates (40M–300M parameters), with poor cross-modal or cross-lingual generalization.

Method

Empirical foundation: The authors first show without any training that extracting hidden-state embeddings from the VLM's own last layers and injecting them at layers 17–19 of the same model (VLM-as-Memory) already outperforms RAG and FastV token pruning. Adding attention-score-based selection of the top-25% embeddings further improves results.

CoMEM architecture: A CLIP retriever fetches the top-10 relevant image-text pairs from WIT (Wikipedia-based Image Text Dataset). Each retrieved pair is passed through the VLM (encoder mode) to obtain continuous representations from the last layers. A shared-parameter Q-Former (k=8 learned query embeddings, L Transformer layers with cross-attention to VLM representations) compresses each item to exactly 8 continuous embeddings, achieving >80× compression over the average 643-token item. The 8×n embeddings are prepended to the inference VLM's input embedding sequence. The inference VLM remains frozen.

Efficient training: LoRA (rank 16) is applied to the VLM encoder; Q-Former parameters are shared across layers. Only these 1.2% of total parameters are trained. Training data (15.6K samples) is self-synthesized: the VLM answers questions in a vanilla RAG setting, retaining only correct outputs (13.8K English from InfoSeek/EVQA/OK-VQA); GPT-4o-mini translates 200 samples into 9 languages for 1.8K multilingual pairs. Training runs on a single H100 GPU in 20 hours, one epoch.

Key Contributions

  • Empirical demonstration that a VLM can serve as its own memory encoder with zero additional training, outperforming RAG and token-pruning baselines.
  • CoMEM: a plug-and-play continuous memory module compressing arbitrary multimodal/multilingual knowledge into 8 embeddings per item via VLM + shared Q-Former.
  • Data- and parameter-efficient training recipe: 1.2% trainable parameters, 15.6K self-synthesized samples, single GPU, 20 hours.
  • Cross-lingual transfer: memory trained on mostly English data generalizes to 9 languages, with +18% on Bulgarian and +10% on Russian InfoSeek.
  • Transferability to pure LLMs: VLM-generated continuous memory improves a language-only Qwen2.5-Instruct from 3.1% to 17.8% average accuracy on visual reasoning benchmarks.

Results

  • Six English multimodal benchmarks (InfoSeek, OVEN, MRAG-Bench, OK-VQA, A-OKVQA, ViQuAE): CoMEM+Qwen2-VL avg. 38.7% vs. Qwen2-VL+RAG 31.1% (+7.6pp); CoMEM+Qwen2.5-VL avg. 35.4% vs. Qwen2.5-VL+RAG 27.6% (+7.8pp).
  • OK-VQA and A-OKVQA: CoMEM achieves >15% absolute improvement over RAG baselines (57.7% and 60.6% on Qwen2-VL vs. 41.9% and 45.3% for RAG).
  • Multilingual InfoSeek (5 languages): CoMEM+Qwen2.5-VL improves overall from 12.5% (RAG) to 24.2% (+11.7pp); CoMEM+Qwen2-VL from 12.1% to 23.4% (+11.3pp).
  • CVQA: CoMEM+Qwen2.5-VL 68.53% vs. RAG 65.76%; CoMEM+Qwen2-VL 67.57% vs. RAG 64.18%.
  • Long-context study: RAG performance degrades past 30 retrieved pairs; CoMEM remains stable up to 50 pairs on InfoSeek.
  • LLM transferability: Qwen2.5-Instruct (text-only) with VLM-generated CoMEM memory: 17.8% avg. (InfoSeek+OVEN) vs. 7.0% with text RAG and 3.1% baseline.
  • CoMEM outperforms or matches specialized models including ReflectiVA (avg. 36.9%) with far less training data and without modifying the inference model.

Limitations

  • Retrieval quality is a hard ceiling: CoMEM depends on CLIP-based retrieval from WIT; poor retrieval yields poor memory regardless of compression quality.
  • Evaluated only on Qwen2-VL and Qwen2.5-VL (7B-scale); generalization to other VLM families not demonstrated in experiments.
  • The multilingual training set is tiny (1.8K translated samples across 9 languages) and generated via GPT-4o-mini translation, potentially introducing translation noise.
  • Compression to exactly 8 embeddings per knowledge item is a fixed design choice; sensitivity to this hyperparameter across longer or more complex knowledge items is not fully explored.
  • Training data self-synthesis filters only correct RAG outputs, which may introduce selection bias toward cases where naive concatenation already works.

Relevance to Vision-Language Models

CoMEM directly addresses a fundamental bottleneck in knowledge-augmented VLMs: the token overhead of multimodal RAG. By showing that a VLM's own intermediate representations are semantically compatible enough to serve as a re-injectable memory for itself, the paper provides both an empirical insight and a practical method that avoids the semantic alignment problem that plagues encoder-decoder memory architectures. The plug-and-play, frozen-inference design is particularly significant for the VLM community, as it enables post-hoc memory augmentation without any modification to deployed models. The cross-modal transferability result—where VLM-encoded continuous memory benefits a text-only LLM—opens a direction for giving language models visual grounding through shared embedding spaces rather than architectural changes.