TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models

🕒 Published (v1): 2026-07-06 01:43 UTC · Source: Arxiv · link

Why this paper was selected

Interpretable concept-overlap token reduction cuts VLM compute with explainable mechanism

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TORINO is a plug-and-play, training-free framework that reduces visual tokens in VLMs by projecting patch embeddings into an interpretable Sparse Autoencoder (SAE) concept space and grouping tokens that share dominant active latents. Unlike attention- or similarity-based methods, grouping is guided by explicit semantic concepts, and the number of retained tokens adapts automatically to per-image complexity without requiring a pre-specified budget.

Problem

Existing visual token reduction methods (ToMe, PruMerge, PruneSID, etc.) assess token importance via proxy signals—attention scores, CLS-similarity, raw embedding distance—that carry no explicit semantic meaning. Tokens that are visually dissimilar but semantically redundant may be incorrectly retained, while tokens encoding the same abstract concept may be kept separately. No prior method exploits the monosemantic structure of SAE features to drive token grouping.

Method

TORINO operates at inference time on a frozen VLM (demonstrated on LLaVA-1.5-7B/13B with CLIP ViT-L/14@336). It proceeds in three stages:

1. SAE encoding. Intermediate patch activations \(\{v_i\}_{i=1}^N\) at a chosen encoder layer are projected through a pretrained Matryoshka BatchTopK SAE: \(z_i = \phi(v_i) \in \mathbb{R}^\omega_{\geq 0}\), with \(\omega = \varepsilon d\) (e.g., \(\varepsilon=64\), \(\omega=65{,}536\)). A peak activation score \(a(i) = \|z_i\|_\infty\) serves as per-token importance.

2. Concept-guided grouping. For each token, the top-\(k\) active concept indices \(A_i = \operatorname{argtopk}(z_i)\) are retained. An undirected graph \(H\) is built where an edge \((i,i')\) exists iff \(|A_i \cap A_{i'}| \geq \delta\). Connected components (found via union-find) define \(G_\text{dyn}\) groups. The pair \((k, \delta)\) controls grouping granularity; \(G_\text{dyn}\) emerges from image content with no explicit complexity estimation.

3. Reduction. Each group is collapsed to one token via:

4. TORINO-P (pruning): retain the token with the highest \(a(i)\).
5. TORINO-M (merging): form a virtual token \(p_g = \frac{1+\log|G_g|}{|G_g|} \sum_{i \in G_g} v_i\), where the log-size rescaling compensates for large, homogeneous groups.

A fixed-budget variant (TORINO\(^\text{FB}\)) enforces exactly \(B\) output tokens by truncating to the \(B\) largest groups and padding with highest-scoring secondary tokens.

Key Contributions

• First SAE-based framework for visual token reduction in VLMs; no fine-tuning required, runs entirely at inference time with frozen weights.
• Concept-guided grouping that captures abstract semantic overlap, not just local appearance similarity.
• A unified pruning/merging framework under the same grouping criterion, enabling controlled comparison of both strategies.
• A dynamic reduction mode where \(G_\text{dyn}\) emerges from image content, plus a fixed-budget variant for batch-processing compatibility.
• Empirical analysis identifying the moderate-compression regime (≈60–80% token reduction) as where semantic grouping yields the largest gains over baselines.

Results

Evaluated on LLaVA-1.5-7B across 9 benchmarks (GQA, MMBench-EN/CN, MME, POPE, ScienceQA, TextVQA, VizWiz, MM-Vet) using VLMEvalKit; relative score = macro-average of per-benchmark ratios vs. full 576-token baseline.

• ∼217 tokens retained (↓62%): TORINO-M achieves 98.7% relative score; TORINO-P 97.1%. Baselines: PruneSID 96.7%, PruMerge 96.7%, FOLDER 95.0%, Random 94.5%.
• ∼129 tokens retained (↓78%): TORINO-P 97.1%, TORINO-M 96.0%. Baselines: PruMerge 94.9%, PruneSID 94.3%.
• ∼47 tokens retained (↓92%): PruneSID leads at 92.8%; TORINO-P and TORINO-M trail at 91.1% and 89.6%.
• Latency (reduction module only, A40 GPU): TORINO-M ≈26 ms vs. PruneSID ≈65 ms and PruMerge ≈71 ms at ∼217-token tier; Random/FOLDER are faster (≈14–17 ms).
• Dynamic variants consistently outperform matched fixed-budget counterparts at moderate reduction, confirming the value of content-adaptive allocation.

Limitations

• Advantage over baselines diminishes at aggressive compression (∼92% reduction, ≤47 tokens), where SAE partitions become too coarse; PruneSID outperforms TORINO there.
• Requires a pretrained SAE matched to the vision encoder's architecture and layer; portability to other encoders demands retraining or adapting the SAE.
• SAE encoding adds non-trivial overhead compared to the simplest baselines (Random, FOLDER), though it remains faster than PruneSID/PruMerge.
• Experiments confined to LLaVA-1.5-7B/13B; generalization to other VLM families (e.g., InternVL, LLaVA-OneVision) is not demonstrated in the main text.
• At very aggressive compression, per-image budgets can drop below 14 tokens for some images, making concept groups too coarse for reliable answering.

Relevance to Vision-Language Models

TORINO directly addresses the visual token bottleneck that constrains inference throughput in standard VLM architectures (LLaVA-style encoder–projector–LLM pipelines), where quadratic attention scaling makes token count the dominant cost factor. It advances the sub-field of training-free token reduction by replacing heuristic proxy signals with mechanistic interpretability tools (SAEs), demonstrating that pragmatic interpretability—repurposing analysis tools for functional decisions—is a viable design axis. The content-adaptive dynamic budget is particularly relevant for researchers interested in input-dependent compute allocation without auxiliary complexity estimators. More broadly, the work bridges mechanistic interpretability of CLIP-style vision encoders with efficient inference engineering, opening a direction where richer semantic representations of token content guide architectural decisions rather than just post-hoc analysis.