Skip to content

QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

QSVD applies Singular Value Decomposition jointly to the concatenated Q, K, V weight matrices of VLM attention blocks, rather than separately, enabling a shared down-projection that reduces KV cache, weights, and FLOPs simultaneously. An adaptive cross-layer rank allocation scores each singular value's importance via first-order loss sensitivity, and post-training quantization (W8A8/W8A4/W4A4) with learnable outlier suppression is applied on top. Combined, the approach outperforms SVD-only and quantization-only baselines by >10% accuracy at lower hardware cost.

Problem

Existing VLM compression methods apply SVD separately to K and V (not Q), or apply quantization alone, leaving significant redundancy unexploited. Per-matrix SVD doubles KV-cache intermediate storage (2rL vs rL) and weight parameters (6rE vs 4rE) compared to a joint decomposition. No prior work combines joint QKV SVD with quantization for VLMs, and naive rank truncation ignores cross-layer importance variation, degrading accuracy unnecessarily.

Method

Step 1 — Joint QKV SVD. Concatenate [Wq, Wk, Wv] ∈ R^{E×3E} into W_concat, apply a single SVD to obtain a shared down-projection W^d_qkv ∈ R^{E×r} and separate up-projections W^u_q, W^u_k, W^u_v ∈ R^{r×E}. During inference, the intermediate C_qkv = X W^d_qkv is buffered (size rL vs 2LE for FP16), and K, V are recomputed on the fly as C_qkv W^u_k, C_qkv W^u_v. Compression is active when r < 0.75E.

Step 2 — Cross-layer rank allocation. For each singular value σ_i, estimate its importance as Î_{σ_i} = E[(σ_i² · (U^T G_W V)_{ii})²] using first-order loss sensitivity on calibration data, requiring only O(E²) memory per layer. All singular values across all layers are globally sorted by importance; the top-k (rank budget) are retained, allocating rank non-uniformly across layers.

Step 3 — Post-training quantization of low-rank VLM. Two orthogonal rotation matrices H1 and H2 suppress channelwise outliers in the input X and the buffered C_qkv. The singular value split exponent β (controlling how singular value magnitudes are distributed between W^d_qkv and W^u_qkv) is learned per-layer over the calibration set to minimize reconstruction error of self-attention output under quantization, directly controlling outlier severity in C_qkv.

Key Contributions

  • Joint SVD over concatenated QKV weight matrix, reducing weight params to 4rE (vs 6rE for per-matrix SVD) and KV-cache buffer to rL (vs 2rL), with shared down-projection enabling single-pass input read during prefill.
  • Cross-layer rank allocation via first-order loss sensitivity scoring (approximation of diagonal Fisher information) computed in O(E²) per layer, with global rank sorting across all attention layers.
  • Quantization scheme adapted for low-rank VLMs: dual orthogonal rotation (H1, H2) for outlier smoothing plus a per-layer learnable exponent β that controls singular value magnitude distribution to minimize quantization error in C_qkv.
  • Demonstration that low-rank approximation reduces VLM hallucination (HallusionBench overall score improves from 26.70 to 30.30 on LLaVA-v1.5 13B at R1=70%).

Results

  • SVD-only (QSVD-noQ) vs. ASVD/SVD-LLM (Table 1): On LLaVA-v1.5 13B, QSVD-noQ achieves 55.79% VizWiz accuracy at R1=46.7%/R2=17.5%, exceeding FP16 (53.63%) and outperforming SVD-LLM (49.37%) under stricter hardware budgets. On SmolVLM 2B, QSVD-noQ exceeds 70% ScienceQA accuracy where ASVD and SVD-LLM fail to operate effectively.
  • Quantized QSVD vs. DuQuant/QVLM/QASVD (Table 2): Under W8A8 at R2=18.75%, QSVD matches FP16 accuracy on LLaVA-v1.5 13B (SciQA: 72.38% vs 71.78% FP16). Under W8A4 at R2=9.38%, QSVD surpasses all baselines. Under W4A4, QASVD yields near-zero accuracy on several models while QSVD remains functional with highest performance.
  • Latency (Figure 4, LLaVA-v1.5 7B on RTX 4070 12GB): QSVD-noQ achieves up to 2.1× speedup over FP16 baseline (which requires CPU offloading); QSVD W8A8 at 50–70% rank retention achieves up to 13.1× speedup by eliminating offloading entirely.
  • Long sequences (HRBench-4K, Table 5): QSVD-noQ consistently outperforms ASVD at matched hardware cost on 4K-resolution inputs.
  • Hallucination (Table 6): LLaVA-Next 13B overall HallusionBench score improves from 36.02 (FP16) to 37.53 (R1=70%); LLaVA-v1.5 13B improves from 26.70 to 30.30.
  • Ablation (Table 3): Cross-layer rank allocation outperforms uniform-rank and Fisher-information-based allocation, especially at aggressive compression.

Limitations

  • SVD and rank allocation are applied only to self-attention QKV layers; FFN layers receive quantization but not joint SVD compression, leaving optimization potential on the table.
  • Calibration uses only 256 samples from ScienceQA training set; generalization of rank allocation and β to other domains is untested.
  • The speedup of 13.1× is measured on a single batch-size-1 scenario with 4K tokens; batched or multi-user inference latency benefits are not characterized.
  • The hallucination reduction effect is noted but causally unconfirmed; the authors acknowledge further investigation is needed.
  • Misuse risks (surveillance, misinformation) from making powerful VLMs more accessible are flagged but not addressed.

Relevance to Vision-Language Models

QSVD directly targets VLM inference efficiency — a critical bottleneck for deploying models like LLaVA on resource-constrained devices — by addressing the KV cache and weight memory jointly rather than optimizing one in isolation. The finding that low-rank QKV compression can reduce hallucination (rather than trade off accuracy for speed) is particularly noteworthy for the VLM community, as it suggests regularization benefits beyond pure compression. The combination of joint SVD with PTQ yields qualitatively different operating points than either technique alone, enabling W8A4/W4A4 regimes where standalone quantization collapses, which is directly relevant to real-time VLM deployment. The cross-layer rank allocation approach generalizes prior Fisher-information methods and could be applied to other structured pruning decisions in multimodal architectures.