Skip to content

Seeing is Free, Speaking is Not: Uncovering the True Energy Bottleneck in Edge VLM Inference

🕒 Published (v1): 2026-07-10 15:31 UTC · Source: Arxiv · link

Why this paper was selected

Reveals decode (LLM text gen) not vision tokens as edge energy bottleneck; inverts prevailing optimization target

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Systematic energy profiling of five VLMs on edge hardware (RTX 3070, Jetson Orin NX) reveals that average inference power is a model-intrinsic constant, so all energy variation reduces to inference time. Autoregressive decoding (output token generation) accounts for 86–97% of total energy, costing 11–39× more wall-clock time per token than prefill, making output length—not visual token count—the true energy bottleneck. Visual token pruning can save at most 10% of total energy for fixed-token models, whereas controlling output length can save up to 97%.

Problem

Existing VLM efficiency work optimizes visual token reduction under the untested implicit assumption that visual processing dominates energy cost on edge devices. No prior work has performed systematic energy profiling of multimodal (vision + language) inference on edge hardware, leaving the true energy breakdown unknown and potentially misdirecting optimization effort.

Method

The authors instrument five VLMs (InternVL3-1B/2B, Qwen2-VL-2B, Qwen2.5-VL-3B, Gemma-3-4B) spanning fixed-token and dynamic-resolution architectures, served via llama.cpp with greedy decoding. Energy is measured on two platforms—RTX 3070 (HWiNFO64 Pro, total system power) and Jetson Orin NX (onboard INA3221 VDD_IN rail)—at 100 ms sampling intervals across four input resolutions (\(224^2\)–\(896^2\)), six image complexity tiers (COCO object-count bins T1–T6), and two prompt types (verbose vs. one-word). They decompose total energy as \(E = \bar{P} \times t\) and fit a linear latency model:

\[t_\text{wall} \approx \alpha_p \cdot N_\text{in} + \alpha_d \cdot N_\text{out} + \beta\]

estimating \(\alpha_p\) (prefill marginal cost per input token) and \(\alpha_d\) (decode marginal cost per output token) via multivariate regression. A five-feature linear energy predictor (model size, \(N_\text{in}\), \(N_\text{out}\), two interaction terms) is also fitted cross-model.

Key Contributions

  • Power fingerprint: Per-inference average power is model-intrinsic (\(\bar{P} = 12.1S + 42.2\), \(R^2=0.918\), where \(S\) is parameter count in billions), invariant to resolution, image content, and prompt type (CV < 5% across all conditions).
  • Energy decomposition: Prefill–decode phase decomposition showing decode accounts for 86–97% of total energy across all models, driven by the compute-bound vs. memory-bandwidth-bound asymmetry (\(\alpha_d / \alpha_p = 11\text{–}39\times\)).
  • Visual token pruning upper bound: Theoretical derivation that removing all visual tokens saves at most 10% of total energy for fixed-token architectures; output length control saves up to 97%.
  • Cross-model energy predictor: A five-feature linear model achieves \(R^2 = 0.986\) of energy variance across all model–resolution–content configurations without per-model calibration.

Results

  • Decode fraction of total inference energy: 86–97% across all five models at 448Ă—448 on RTX 3070.
  • Per-output-token cost ratio (\(\alpha_d/\alpha_p\)): 11Ă— (InternVL3-1B) to 39Ă— (Qwen2-VL-2B); see Table 2.
  • Image complexity (T1 vs. T6, 1 vs. 12–16 objects): up to 4.1Ă— energy difference at identical resolution, entirely attributable to output length variation, not visual processing cost.
  • Dynamic-resolution models (Qwen2.5-VL-3B): 8–24% energy increase from \(224^2\) to \(896^2\); fixed-token models (InternVL3, Gemma-3): virtually no energy change across resolutions.
  • At \(896^2\), Qwen2.5-VL-3B reaches prefill–decode parity (55% vs. 45%) due to 14Ă— more visual tokens but similar output length—additional prefill is largely wasted.
  • Output length control: saves up to 97% of total energy; visual token pruning upper bound ≤ 10% for fixed-token models.
  • Cross-model linear predictor: \(R^2 = 0.986\) without per-model calibration.

Limitations

  • Study is confined to quantized (Q4_K_M/Q8_0) models served via llama.cpp; results may not generalize to full-precision inference or other serving stacks (e.g., TensorRT-LLM, vLLM).
  • Only two hardware platforms tested; results for mobile SoCs, NPUs, or specialized accelerators may differ.
  • Output length is treated as an observed variable, not controlled; the paper does not propose a concrete mechanism to reduce output length while preserving task quality.
  • Model scale limited to 1B–4B parameters; scaling behavior toward 7B+ on edge is extrapolated, not measured.
  • Power measurement uses system-level rails (total system, not GPU-only), which may introduce platform-specific noise.

Relevance to Vision-Language Models

This paper directly challenges a foundational assumption in VLM efficiency research—that visual token reduction is the primary lever for energy savings—by providing the first systematic empirical energy decomposition of on-device VLM inference. For researchers working on efficient VLMs, the finding that decoding dominates energy (86–97%) shifts the optimization target from vision encoders and visual token pruning to output length control (early stopping, constrained generation, structured outputs). The cross-model energy predictor with \(R^2=0.986\) also provides a practical tool for energy-aware deployment decisions without hardware profiling. This reframes the design space for edge-deployable VLMs in embodied AI, suggesting that future architectures should co-optimize visual encoding efficiency with verbosity of generated responses.