Skip to content

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

🕒 Published (v1): 2026-05-19 17:50 UTC · Source: Arxiv · link

Why this paper was selected

Mix-Quant: quantized prefilling with precise decoding cuts agentic LLM input overhead

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Mix-Quant is a phase-aware quantization framework for LLM agent inference that applies NVFP4 (4-bit floating-point) quantization exclusively during the prefilling phase while retaining BF16 precision for autoregressive decoding. This decoupling exploits the observation that prefilling exhibits high quantization redundancy (due to attention mass concentration and input fixedness), while decoding is sensitive to error accumulation over long trajectories. The result is up to 3Ă— prefill speedup with minimal task performance degradation.

Problem

Agentic LLM workflows are severely input-heavy: multi-turn tool use, memory retrieval, and environment feedback cause input contexts to grow to tens–hundreds of times the output length, making the compute-intensive prefilling stage the dominant inference bottleneck. Existing quantization methods either target only weights (improving decoding bandwidth but not prefill compute) or apply uniform weight-and-activation quantization to all phases—causing significant accuracy degradation in long agentic trajectories due to error accumulation during decoding.

Method

Mix-Quant maintains two execution paths over the same base model:

  1. FP4 prefilling path: applies NVFP4 W4A4 (weight-and-activation) microscaling quantization to the entire context encoding stage. NVFP4 uses E2M1 4-bit values with two-level scaling: per-block FP8 E4M3 scales over groups of \(g=16\) elements, plus a tensor-level scale \(\alpha_x\). Quantization follows: $\(\hat{x}_i = \alpha_x \sigma_{b(i)} q_i, \quad q_i = \Pi_{\text{FP4}}\!\left(\frac{x_i}{\alpha_x \sigma_{b(i)}}\right)\)$ with block scales set by \(\sigma_b = \Pi_{E4M3}\!\left(\frac{\max_{i\in b}|x_i|}{\alpha_x q_{\max}}\right)\). Round-to-nearest (RTN) is used directly (no rotation needed at this format).

  2. BF16 decoding path: autoregressive token generation runs at full BF16 precision, consuming the KV cache written by the FP4 prefill path.

These paths are deployed using a prefill-decode disaggregation framework (NIXL-based KV-cache transfer between prefill workers and decode workers), avoiding kernel-switching or cache-alignment overhead. The key theoretical justification for safe FP4 prefilling is attention mass concentration: in a 128K-token context, the top-4,096 tokens (3.125% of context) account for 95.8% of total attention mass, so quantization errors on low-attention tokens are heavily attenuated.

Key Contributions

  • Empirical characterization of agentic workflows as input-heavy: prefilling, not decoding, is the dominant bottleneck in multi-turn agent inference.
  • Identification that FP4 prefilling is safe (redundancy due to context fixedness and attention concentration) while FP4 decoding causes error snowballing over long trajectories.
  • Mix-Quant: a phase-aware W4A4/BF16 split framework leveraging NVIDIA Blackwell NVFP4 hardware with a disaggregated serving architecture.
  • Empirical validation across 4 models (Qwen3-8B, Qwen3.5-9B, Gemma-4-26B-A4B-it, Gemma-4-31B-it) on 5 benchmarks (BFCL v4, LongMemEval, τ²-bench, LongBench-V2, AA-LCR, MATH500, AIME24/25).

Results

  • Prefill speedup: 2–3.74Ă— over BF16 baseline on RTX 5090, across sequence lengths and batch sizes.
  • Agentic benchmarks (vs. uniform NVFP4 baseline):
  • Qwen3-8B: 41.45 (Mix-Quant) vs. 38.64 (NVFP4) vs. 42.85 (BF16)
  • Qwen3.5-9B: 74.68 vs. 70.37 vs. 77.31
  • Gemma-4-26B-A4B-it: 61.67 vs. 55.95 vs. 66.07
  • Gemma-4-31B-it: 77.14 vs. 76.21 vs. 77.63 (near-BF16 recovery)
  • LongMemEval (most sensitive benchmark):
  • Qwen3-8B: Mix-Quant 54.85 vs. NVFP4 49.82
  • Gemma-4-26B: Mix-Quant 72.45 vs. NVFP4 62.42
  • Reasoning (AIME24, Qwen3-8B): Mix-Quant 76.66 vs. NVFP4 66.53 vs. BF16 75.54 (Mix-Quant slightly exceeds BF16)
  • Long-context (Gemma-4-26B, averaged): Mix-Quant 71.93 vs. BF16 71.94 — essentially lossless.

Limitations

  • Requires NVIDIA Blackwell hardware (RTX 5090, B200) for NVFP4 tensor-core acceleration; not portable to older GPU generations.
  • Mix-Quant does not recover full BF16 performance on all models/benchmarks (e.g., Qwen3.5-9B τ²-bench: 81.89 BF16 vs. 81.89 → 76.26 NVFP4 → 81.89... actually 81.89 vs. 81.89 — still a gap of ~5pp on some tasks like LongMemEval for Qwen3.5: 86.27→84.27).
  • KV-cache transfer via NIXL between disaggregated prefill and decode workers adds system complexity and network/memory overhead not quantified in the paper.
  • Evaluation limited to open-weight models up to ~31B parameters; behavior at much larger scales (70B+) is not validated.
  • The RTN quantization approach may underperform more sophisticated calibration-based methods; this trade-off is acknowledged but not fully explored.

Relevance to Harnesses / Meta-Harnesses

Agentic harnesses and meta-harnesses are the primary deployment target of Mix-Quant: the paper explicitly models the multi-turn, multi-tool-call loop as the workload, where repeated context accumulation (system prompt + tool schemas + retrieved memory + prior trajectories) makes prefilling the bottleneck. Phase-aware quantization is a systems-level optimization that harness developers must expose or configure—specifically, whether the serving backend applies FP4 to prefill workers and BF16 to decode workers—making this directly relevant to anyone building efficient long-horizon agent infrastructure. The prefill-decode disaggregation architecture (separate prefill/decode workers with KV-cache handoff) is a design pattern that harness orchestrators must account for when scheduling multi-agent or multi-turn pipelines. For meta-harnesses that orchestrate many concurrent agents, the 2–3× prefill speedup translates directly to higher agent throughput and reduced latency per reasoning step.