Skip to content

Look Less, Think Faster: Joint Token-Compute Adaptation for Multimodal LLMs

🕒 Published (v1): 2026-07-22 16:43 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; joint token-compute pruning cuts MLLM inference cost; directly deployable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SmartVL is an adaptive inference framework for Multimodal LLMs that jointly optimizes three computational axes — visual token count, transformer layer depth, and attention head width — conditioned on both input content and a runtime compute budget. Unlike prior work that prunes tokens or skips layers in isolation, SmartVL couples these decisions end-to-end via a shared budget encoding and a differentiable FLOPs estimator. At ~50% compute budget, SmartVL outperforms prior adaptive methods by an average of 7.8% across seven benchmarks.

Problem

MLLMs incur a fixed inference cost regardless of input complexity, and prior efficient-inference methods address computational redundancy along a single axis (token pruning or layer/head skipping) independently. This decoupling creates stage incompatibility: isolated token pruning discards spatially redundant but semantically critical tokens without awareness of downstream LLM needs, while isolated layer skipping assumes full-length token sequences and cannot adapt when upstream pruning has already simplified context. No existing method dynamically co-allocates compute across all three axes (sequence length \(S\), depth \(L_{\text{eff}}\), width \(\alpha_l\)) as a function of both input content and runtime budget \(b\).

Method

SmartVL decomposes the adaptive allocation policy \(\pi\) into two lightweight, jointly trained controllers linked by a shared budget signal:

Vision-side token controller. A scalar budget \(b\) is sinusoidally encoded, projected via an MLP, and appended as a budget token to the ViT input sequence. After attending over all image patches, this token produces a content- and budget-aware state \(\mathbf{h}_b\). Per-token binary masks \(m_{\text{token}} \in \{0,1\}^N\) are generated via Gumbel-sigmoid with a straight-through estimator (STE): $\(y_i = \sigma\!\bigl((z_i + \xi_i)/\tau\bigr), \quad m_i = \mathbf{1}[y_i > 0.5] - \text{sg}(y_i) + y_i\)$ This enables differentiable training over discrete token selection. At inference, noise is removed and tokens with \(\sigma(z_i) \leq 0.5\) are physically discarded; RoPE handles the compressed sequence natively.

LLM-side compute controller. After \(P\) mandatory prefix layers, the compute controller receives the fused state \(\tilde{\mathbf{h}}_b^{(P)} = \mathbf{h}_b^{(P)} + \mathbf{e}_\kappa\), where \(\mathbf{e}_\kappa\) encodes the normalized token survival rate \(\kappa = \text{sg}(\sum_i \sigma(z_i))/N\). The stop-gradient on \(\kappa\) decouples the two controllers' optimization trajectories, preventing gradient magnitude imbalance. Independent Gumbel-sigmoid activations per layer (Layer controller, L) or per head-group per layer (Layer-Head controller, LH) produce binary masks \(m_{\text{layer}}\) and \(m_{\text{head}}\); the active fraction is \(\alpha_l = \sum_j M_{l,j}/G\).

Joint training uses a differentiable FLOPs estimator: $\(\hat{\mathcal{C}} = \sum_{l=1}^{L} \alpha_l \bigl[(8D^2 + 6DD_{\text{int}}) \cdot S + 4D \cdot S^2\bigr] + 2DV \cdot S\)$ An asymmetric budget violation loss quadratically penalizes over-budget usage and linearly penalizes under-utilization within a tolerance margin \(\mu\): $\(\mathcal{L}_{\text{over}} = \max(0, r - b)^2, \quad \mathcal{L}_{\text{under}} = \max(0, (b - \mu) - r)\)$ with a linear warmup schedule \(\lambda(t) = \min(1, t/T_{\text{warmup}})\) to prevent premature degenerate sparsity. Budget \(b\) is sampled uniformly during training for generalization across the compute spectrum.

Key Contributions

  • Joint three-axis adaptive inference framework (sequence length, depth, width) that explicitly models the coupling between visual token density and LLM architectural capacity.
  • Shared budget encoding connecting both controllers, enabling cross-stage allocation decisions conditioned on input content and runtime budget without manual threshold tuning.
  • Differentiable FLOPs estimator providing an end-to-end gradient path through discrete routing decisions, combined with Gumbel-sigmoid sampling and asymmetric budget violation loss for stable joint optimization.
  • Broader adaptation range: SmartVL operates from 2.5T to 8.5T FLOPs versus AdaLLaVA-PruMerge's 3T–4.8T window.

Results

  • At ~50% compute budget, SmartVL outperforms AdaLLaVA by 6.6% accuracy on VQAv2 and 7.8% average across seven benchmarks (VQAv2, GQA, TextVQA, ScienceQA, POPE, MMBench, VizWiz).
  • SmartVL achieves a superior accuracy–efficiency Pareto frontier over all individual-axis baselines (FastV, LLaVA-PruMerge+) and the naive composed baseline (AdaLLaVA-PruMerge) across all tested FLOPs levels.
  • Within the 3T–4.8T window, SmartVL delivers comparable accuracy to LLaVA-PruMerge+ while scaling across a much wider operating range.
  • Consistent improvements hold from the minimum compute budget up to full model capacity.

Limitations

  • The paper evaluates on a specific MLLM family (LLaVA-style, MLP projection); generalization to QFormer-based architectures (BLIP-2, InstructBLIP) or natively sparse encoders (FastVLM) is not demonstrated.
  • FLOPs are used as a hardware-independent budget proxy; actual wall-clock latency gains under real deployment constraints (memory bandwidth, KV-cache, batching) are not fully characterized.
  • The stop-gradient on \(\kappa\) decouples controller gradients by design, which may limit the degree to which the two controllers truly co-optimize in a globally optimal sense.
  • Budget is applied uniformly at the batch level during training, which may limit fine-grained per-instance adaptation compared to instance-level budget scheduling.
  • The paper text provided is truncated before full ablation results are reported.

Relevance to Vision-Language Models

SmartVL directly targets the inference efficiency bottleneck of the dominant MLLM architecture (ViT encoder + projection + LLM), which is critical as VLMs scale to larger backbones and higher-resolution inputs. The joint token-compute co-adaptation paradigm challenges the prevailing design assumption that visual token pruning and LLM compression can be independently optimized, offering a principled coupling mechanism that is broadly applicable to any patch-token-based VLM. The shared budget encoding and differentiable latency estimator provide a reusable training recipe for elastic VLM deployment — relevant to real-time vision-language applications under variable compute constraints. This work establishes a stronger Pareto baseline against which future efficient-VLM methods should be measured.