VCM: Vision Concept Modeling with Adaptive Vision Token Compression via Instruction Fine-Tuning¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VCM introduces a self-supervised framework for vision concept modeling in large vision-language models (LVLMs), enabling dynamic extraction of task-relevant visual tokens based on instruction priors rather than processing all tokens uniformly. The core mechanism combines a keyword selector module with a CTC-style forward-backward dynamic programming algorithm to adaptively determine both the number and spatial positions of retained vision concepts. VCM achieves up to 85% FLOP reduction on LLaVA-1.5-7B while improving performance on dense perception tasks beyond vanilla VQA.

Problem¶

Current LVLMs process entire images at the token level, generating large numbers of vision tokens (e.g., 576 for LLaVA-1.5) regardless of task specificity. This creates quadratic attention cost that bottlenecks high-resolution image and long video processing. Prior token compression methods (FastV, SparseVLM, VisionZip, PruMerge) prune tokens without preserving semantically meaningful spatial alignment, making them unsuitable for dense perception tasks such as open-vocabulary detection and segmentation. No prior method defines a proper vision concept model that outputs both the quantity and spatial locations of concepts conditioned on instructions.

Method¶

Empirical motivation. The authors sample 5K LLaVA instruction-tuning instances, use GPT-4o to count image-related keywords in instructions vs. responses, and show a stable negative correlation between (response_keywords − instruction_keywords) and the minimum required token length. Vision concept length is estimated from this difference via min-max normalization, scaled by √2 after local merging.

Keyword Selector Module (KSM). A multi-head self-attention (MHSA) layer computes a global text feature from instruction+response tokens, then cross-attends against global CLIP vision features to score individual tokens. Keywords with scores above the mean are retained. In pre-training, a semantic alignment (SA) loss aligns global vision and text features in vocabulary space.

Forward-Backward Dynamic Programming. Given M aligned vision tokens and a target concept length L (estimated from keyword difference), a binary classification head assigns each token a keep/skip probability. The alignment problem is formulated as CTC (Connectionist Temporal Classification): a target sequence of L retained symbols plus blanks of length 2L+1 is defined, and forward (α) and backward (β) probabilities are computed in O(M²) to yield the VCM loss LVCM = −log p(Z^V|Y^V). Gradients flow back to the classification head. Retained tokens are merged via a segment merging (SM) operation using weighted averaging by importance scores.

Training. Two-stage: (1) pre-training with SA loss + next-token prediction, (2) instruction fine-tuning with LVCM + next-token prediction loss, weighted by a hyperbolic-tangent coefficient ϵ(r) that scales with mask ratio r. During training, instruction keywords are randomly masked (ratio r ~ Uniform(0,1)) as implicit contrastive sampling; higher masking forces learning of longer concept sequences.

Key Contributions¶

Formal definition of the vision concept model: a module that dynamically determines vision concept count and spatial locations from instruction priors.
VCM framework: self-supervised learning from coarse VQA data without concept-level annotations, leveraging vision-language correlations.
CTC-derived forward-backward algorithm for variable-length vision token selection with O(M²) complexity and theoretical gradient support.
Demonstration that vision concept modeling improves both efficiency (fewer FLOPs) and dense perception capability (classification, detection, segmentation) of the CLIP ViT backbone.

Results¶

VQA (11 benchmarks, 144 tokens vs. 576 for LLaVA-1.5-7B): VCM achieves 60.8% avg vs. 59.5% for VisionZip (192 tokens), 57.8% for PDrop (192 tokens), 57.9% for SparseVLM (192 tokens). VCM outperforms the vanilla model on avg with fewer tokens.
FLOPs: Up to 85% reduction for LLaVA-1.5-7B.
Region-level VQA (RefCOCO, ViT-L/14): VCM improves testA BBox from 40.1 → 41.3, testB from 29.5 → 31.8.
Zero-shot image classification (COCO, ViT-L/14 thing masks): Top-1 28.3 → 43.8; Top-5 52.0 → 69.7.
OV detection (OV-COCO, ViT-L/14): AP^novel_50 9.2 → 12.6; AP^base_50 44.3 → 48.7; AP_50 35.2 → 39.2.
OV segmentation (ADE-150, Cat-Seg + ViT-L/14): mIoU 32.1 → 34.8; mAcc 50.7 → 53.1.

Limitations¶

The target concept length L is estimated via a heuristic min-max normalization from keyword statistics derived using GPT-4o; errors in keyword counting propagate directly to token budget.
Dynamic programming complexity is O(M²) per sample, which may become a bottleneck for very high-resolution inputs with large M despite reducing downstream LLM cost.
Experiments are confined to the LLaVA-1.5 / LLaVA-NeXT / Video-LLaVA family; generalization to other LVLM architectures is deferred to appendix experiments only.
Random masking of keywords as implicit contrastive sampling requires accurate keyword identification, which depends on MHSA-based selection quality at inference time.
The method requires two-stage re-training of the projector and classification head; it cannot be applied purely at inference time unlike some pruning approaches.

Relevance to Vision-Language Models¶

VCM directly addresses one of the central efficiency bottlenecks in scaling LVLMs to high-resolution or video inputs by proposing a principled, instruction-conditioned token selection mechanism — going beyond static pruning to concept-level representation. Unlike prior token compression work that degrades dense perception tasks, VCM's spatial alignment preservation makes it directly applicable to grounding, detection, and segmentation pipelines that share a CLIP backbone. The CTC-inspired training objective is a novel reuse of a sequence-alignment technique from ASR, offering a theoretically grounded alternative to attention-score heuristics. For researchers tracking VLMs, this paper offers a clean formalism (vision concept model) and a reusable self-supervised training recipe applicable to any LVLM built on a CLIP ViT backbone.