WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; weighted SVD for fast low-precision VLM execution

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

WSVD applies SVD at per-head granularity rather than across full K/V projection matrices, eliminating the memory-access bottleneck that makes conventional SVD-based compression counter-productive at inference time. It further incorporates element-wise Fisher-weighted fine-tuning and quantization-aware training (QAT) with Hadamard rotation for outlier smoothing. The result is >1.8× decoding speedup over full-precision baselines while preserving benchmark accuracy.

Problem¶

Prior SVD-based KV compression methods for VLMs (ASVD, SVD-LLM, QSVD) store shared low-rank latents $C_K = XA_K \in \mathbb{R}^{L \times R}$ and reconstruct per-head keys as $K_h = C_K B_{Kh}$. In practice this requires loading the full shared latent for every head at every decoding step, so effective memory traffic per head is $\eta_\text{svd} = LR$ — often higher than storing the original K cache. Measured on LLaVA-Next 7B (RTX 4090, batch 16, KV length 8192), the SVD baseline is slower than uncompressed flash decoding, nullifying the compression benefit.

Method¶

Per-head SVD (§3.1). Instead of decomposing the global $W_K \in \mathbb{R}^{E \times E}$, WSVD decomposes each head's submatrix $W_{Kh} \in \mathbb{R}^{E \times H}$ as $W_{Kh} = A_{Kh} B_{Kh}$ with $A_{Kh} \in \mathbb{R}^{E \times r}$, $B_{Kh} \in \mathbb{R}^{r \times H}$. Each head caches its own latent $C_{Kh} = XA_{Kh} \in \mathbb{R}^{L \times r}$; since $H \ll E$, $r \ll R$, reducing per-head memory traffic and compute by a factor of $r/R$.

Weighted fine-tuning (§3.2). Standard SVD ignores that "superweights" dominate accuracy. WSVD computes element-wise Fisher importance scores $F_{Kh} = \mathbb{E}_{x \sim \mathcal{D}}[g_{Kh}(x) \odot g_{Kh}(x)]$ and solves: $$\min_{A_{Kh}, B_{Kh}} \sum_h \left\| F_{Kh}^{1/2} \odot (W_{Kh} - A_{Kh}B_{Kh}) \right\|_F^2$$ locally per head, preserving high-sensitivity weights during truncation.

Quantization-aware training (§3.3). Hadamard matrix $S_1$ and learnable rotation $S_2$ are inserted to smooth activation outliers. The transformed QAT objective is: $$\min_{S_2, A_h, B_h} \left\| (F_h')^{1/2} \odot \left( S_1 W_h - Q(S_1 A_h S_2^\top) Q(S_2 B_h) \right) \right\|_F^2$$ jointly updating $A_h$, $B_h$, $S_2$ while $S_1$ stays fixed.

Fused Triton kernel (§3.4). A custom Flash-Decoding kernel streams $C_{Kh}$ tiles on-chip, reconstructs $K_{h,t} = C_{Kh,t} B_{Kh}$ in registers, and immediately contracts with $q_h$ — eliminating full-$K$ materialization in VRAM. $B_{Vh}$ is fused into the output projection to avoid explicit $V_h$ reconstruction.

Key Contributions¶

Per-head SVD that reduces per-head latent memory traffic from $\eta_\text{svd}=LR$ to $\eta_\text{wsvd}=Lr$ (ratio $r/R$), enabling real decoding latency reduction.
Element-wise Fisher-score-weighted local fine-tuning to compensate accuracy loss amplified by finer-grained decomposition.
Quantization-aware training with Hadamard + learnable rotation applied jointly to the low-rank factors, targeting both weights and activations.
Triton fused kernel integrating per-head low-rank reconstruction, RoPE, $qK^\top$ accumulation, softmax, and $V$ multiplication within a single flash-decoding pass.

Results¶

Decoding speedup: >1.8× over full-precision Flash Decoding baseline across LLaVA-v1.5 7B/13B, LLaVA-Next 7B/13B, SmolVLM-Instruct 2B.
ScienceQA-IMG accuracy (FP16, $\rho_1 = 50\%$): WSVD-noQ 65.89% vs. QSVD-noQ 65.05%, SVD-LLM 57.41%, ASVD 19.19% on LLaVA-v1.5 7B.
SEED-Bench-IMG (LLaVA-Next 13B, $\rho_1 = 50\%$): WSVD-noQ 60.46% vs. QSVD-noQ 58.23%, SVD-LLM 55.31%, ASVD 24.17%.
WSVD-noQ consistently outperforms all SVD baselines (ASVD, SVD-LLM, QSVD-noQ) at every compression ratio tested ($\rho_1$ from 90% down to 50%).
Combined SVD + QAT variant (WSVD) further narrows the gap to or exceeds FP16 accuracy on several settings.

Limitations¶

Calibration set is small (256 ScienceQA training samples); generalization of Fisher importance estimates to out-of-domain tasks is unverified.
Per-head SVD increases approximation error relative to full-matrix SVD, which the weighted fine-tuning only partially compensates — error control at very aggressive compression ratios remains challenging.
Evaluation is limited to five LLaVA-family / SmolVLM models; applicability to architectures with different attention designs (e.g., grouped-query attention, MQA) is not demonstrated.
The custom Triton kernel adds engineering complexity and hardware dependency (GPU with sufficient on-chip SRAM); CPU or mobile deployment is not evaluated.
Results truncated in the provided text; quantized WSVD numbers and full SEED-Bench results are partially in the appendix, limiting full comparison.

Relevance to Vision-Language Models¶

WSVD directly targets the inference bottleneck of VLMs — memory-bandwidth-bound KV-cache access — with a compression scheme that achieves real-hardware speedup rather than theoretical FLOP reduction. For VLM researchers, this is practically significant: it demonstrates that standard SVD applied at the wrong granularity can degrade latency, a finding that reframes prior compression results in the literature. The Fisher-weighted QAT framework is model-agnostic and can compose with other VLM efficiency techniques (e.g., token pruning, speculative decoding). The open-sourced Triton kernel also provides a concrete systems artifact for efficient multimodal inference.