Skip to content

Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper derives inference-time scaling laws for VLMs that jointly model LLM parameter count and visual token count as the two levers of inference cost. The central finding is that, under a fixed inference FLOP budget, visual reasoning tasks are compute-optimally served by maximizing LLM size while aggressively compressing visual tokens — often to a single token. The authors then propose a query-conditioned token compression method tailored for these extreme compression regimes.

Problem

Prior work on visual token compression targets moderate reductions (576→144 or 64 tokens) while trying to preserve base-model accuracy, but no principled framework existed to determine the optimal trade-off between LLM size and visual token count under a fixed inference budget. It was unclear whether, e.g., a 4B LLM with 576 tokens outperforms an 8B LLM with 288 tokens at the same FLOP cost.

Method

The authors formulate a two-factor scaling law for VLM downstream error:

\[Y(N, T) = \frac{A}{N^\alpha} \cdot \frac{B}{T^\beta} + D\]

where \(N\) is LLM parameter count, \(T\) is total inference tokens (visual + text), and \(\{\alpha, \beta, A, B, D\}\) are fit via grid search. They train LLaVA-Next VLMs with Qwen-1.5 backbones ({0.5, 1.8, 4, 7}B) using TokenPacker-compressed visual tokens ({1, 4, 16, 36, 64, 144, 576}) and evaluate on nine visual reasoning benchmarks to fit the law, then validate predictions on a held-out 14B model. A modified TokenPacker variant incorporating the user query prompt (cross-attention over query tokens during compression) is introduced to handle extreme compression (1–16 tokens) without catastrophic loss.

Key Contributions

  • First inference-time compute-optimal scaling laws for VLMs jointly parameterized by LLM size and visual token count.
  • Empirical finding that \(\alpha \approx 0.077 \gg \beta \approx 0.015\) for visual reasoning: error degrades 5× faster per unit of LLM parameter reduction than per unit of visual token reduction, making token compression strictly preferable to LLM downsizing.
  • Demonstration that the Pareto-optimal inference curve (Q=0) collapses to a single visual token paired with the largest feasible LLM; under Q=50 text tokens, the optimum shifts to ~16 visual tokens (97% reduction from 576).
  • Contrasting result for OCR/document tasks: \(\alpha = 0.029 < \beta = 0.048\), so token count dominates and compression is harmful.
  • Query-aware TokenPacker variant that mitigates performance gaps at 1- and 4-token budgets (e.g., 34% gap reduction on VizWiz, ~12% on POPE and MMBench vs. standard TokenPacker).
  • Scaling law generalization verified across a second compression algorithm (LLaVA-PruMerge).

Results

  • Scaling law predicts 14B VLM performance (extrapolated from 0.5–7B fits) within <2% error margin.
  • At matched inference FLOP budgets, switching from 1.8B LLM + 144 tokens → 7B LLM + 36 tokens yields significant relative gains on visual reasoning tasks (GQA, MMBench, MME, MMMU, POPE, ScienceQA), while harming text recognition (DocVQA, TextVQA).
  • Query-conditioned compression at 1 token: closes 34% of the VizWiz gap and ~12% of POPE/MMBench gap vs. vanilla LLaVA-1.5.
  • Repeating visual tokens multiple times does not improve performance, ruling out test-time compute scaling via token repetition.
  • PruMerge-based scaling laws yield \(\alpha = 0.069\)–\(0.077\), \(\beta = 0.008\)–\(0.041\), confirming the same qualitative conclusion across architectures.

Limitations

  • Scaling laws are fit on Qwen-1.5 family and a single CLIP-ViT-L encoder; generalization to other encoder families (e.g., SigLIP, DINOv2) or dynamic-resolution encoders is untested.
  • Results hold for short-response visual reasoning and OCR benchmarks; tasks requiring long-form generation (G ≫ 0) are explicitly excluded from the analysis.
  • Query-conditioned compression requires the user query at compression time, which may not be available in all deployment pipelines (e.g., cached/static visual features).
  • Adaptive in-LLM token pruning methods are explicitly left out of scope.
  • Token repetition as a test-time scaling mechanism was only evaluated on current (not future, harder) benchmarks; the conclusion may not generalize to more reasoning-intensive VLM tasks.

Relevance to Vision-Language Models

This paper reframes the token compression problem in VLMs from "preserve accuracy while reducing tokens modestly" to "find the compute-optimal operating point," which has direct practical implications for deploying VLMs on edge devices. The finding that visual reasoning tasks tolerate extreme token compression (1–16 tokens) far better than LLM parameter reduction inverts the current community focus and motivates a new class of high-compression projector designs. The task-type bifurcation (reasoning vs. OCR) is an important caution: architecture and compression choices need to be task-aware. For researchers tracking VLMs, these scaling laws provide a rigorous quantitative foundation — analogous to Chinchilla for training — but applied to inference efficiency.