Skip to content

SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpaceServe is an MLLM serving system that replaces sequential (time-multiplexed) encoder→decoder GPU scheduling with simultaneous (space-multiplexed) co-location by partitioning streaming multiprocessors (SMs) at fine granularity. It exploits the complementary resource profiles of compute-bound vision/audio encoders and memory-bound text decoders to reduce time-per-output-token by up to 28.9× over vLLM v1 on A100 GPUs.

Problem

State-of-the-art MLLM serving stacks (vLLM, TGI) treat encoders and decoders as sequential jobs: the GPU runs the vision/audio encoder, then the text decoder, wasting the complementary compute/memory footprints of each. Encoder kernels are compute-bound (arithmetic intensity up to 8826 FLOPs/byte at 2K resolution), while decoder autoregression is memory-bound (∼0.887 FLOPs/byte), so sequential scheduling idles SMs during decoding and squanders memory bandwidth during encoding.

Method

SpaceServe introduces three interlocking mechanisms:

  1. Split-encoder architecture: Modality encoders are fully decoupled from the LLM decoder into separate processes (CPU process, Encoder process, LLM process), enabling independent scheduling.

  2. Space Inference Scheduler (SIS): Offline hierarchical profiling builds Resource–Utility Curves mapping SM count → latency for every key operator type (GEMM, attention) across encoder and decoder variants at varying resolutions. At runtime, SIS uses these curves to find the SM partition that minimises the sum of encoder + decoder execution times, dynamically re-partitioning per incoming request using hardware primitives (libsmctrl on NVIDIA, CU masks on AMD).

  3. Time-Windowed Shortest-Remaining-First (TWSRFT) encoder scheduler: Within each fixed time window, encoder requests are batched by ascending patch count (smallest-first) while respecting a per-GPU batch capacity ceiling (seqlengthbatchsuitable). This minimises encoding tail latency and smooths the token arrival rate seen by the decoder.

Key Contributions

  • First serving system to decouple MLLM modality encoders from the shared text decoder and co-locate their kernels on the same GPUs via sub-GPU SM partitioning.
  • Cost-model-guided SIS: offline operator-level profiling aggregated into end-to-end Resource–Utility Curves, enabling adaptive per-request SM allocation.
  • TWSRFT scheduler that minimises encoder completion latency under bursty mixed-modality workloads without starving large-image requests.
  • Demonstrated up to 28.9× TPOT reduction over vLLM v1 across four MLLM sizes (2B–72B) and generalises to MoE-based architectures (DeepSeek-VL2, Kimi-VL).

Results

  • Qwen2-VL-2B (1×A100, 10 RPS): SpaceServe TPOT 12.62 ms vs. vLLM 365 ms → 28.9× speedup.
  • Qwen2-VL-7B (1×A100, 10 RPS): 40 ms vs. 489 ms → 12.3× speedup.
  • Qwen2.5-VL-32B (4×A100, 10 RPS): 144 ms vs. 522 ms → 3.62× speedup.
  • Qwen2-VL-72B (8×A100, 10 RPS): 155.8 ms vs. 489 ms → 3.14× speedup.
  • MoE models (10 RPS): DeepSeek-VL2 4.08×, Kimi-VL 9.84× TPOT speedup over vLLM v1.
  • Resolution scaling (Qwen2-VL-7B): TPOT speedup rises from 1.37× at 224×224 to 12.39× at 2K×2K.
  • Average TPOT reduction: 4.81× across all tested configurations.
  • TTFT: Virtually unchanged relative to vLLM (SpaceServe targets steady-state decode throughput, not first-token latency).
  • Ablation – SIS vs. NVIDIA MPS (Qwen2-VL-7B, 10 RPS): SIS achieves 39.8 ms vs. MPS 132 ms → 3.3× advantage due to SM-level isolation eliminating intra-SM resource contention.
  • Ablation – TWSRFT vs. vLLM scheduler (10 RPS): 39.8 ms vs. 114.22 ms.

Limitations

  • TTFT is not improved; SpaceServe only accelerates steady-state decoding throughput.
  • Offline profiling overhead: Resource–Utility Curves must be rebuilt for new model architectures or GPU types.
  • SM partitioning relies on libsmctrl/CU masks, which are hardware-specific APIs that may not be uniformly available or stable across all GPU generations and drivers.
  • Evaluation uses only NVIDIA A100 GPUs; generalization to other GPU families (H100, consumer GPUs) is not empirically demonstrated in the main text.
  • Multi-GPU configurations use tensor parallelism; interaction with pipeline parallelism or more complex parallelism strategies is unaddressed.

Relevance to Vision-Language Models

SpaceServe directly addresses a core deployment bottleneck for VLMs: the encoder-induced stall during autoregressive decoding. As VLMs push toward higher-resolution inputs (the paper shows encoder AI growing from 338 to 8827 FLOPs/byte from 512→2K), this bottleneck becomes proportionally worse, making space-multiplexing increasingly critical. The paper also shows applicability to MoE-based VLMs (DeepSeek-VL2, Kimi-VL), where sparse decoder activations amplify resource heterogeneity. For researchers studying efficient VLM inference, SpaceServe represents a systems-level counterpart to model-level efficiency work (token pruning, KV compression), addressing the serving stack rather than the architecture itself.