ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ElasticMM is an MLLM serving system that introduces Elastic Multimodal Parallelism (EMP), decoupling text-only and multimodal requests into separate modality groups and elastically adjusting per-stage parallelism at runtime. By combining modality-aware load balancing, elastic partition scheduling, unified multimodal prefix caching, and non-blocking encoding, it reduces TTFT by up to 4.2× and achieves 3.2–4.5× throughput gains over vLLM while meeting SLOs.

Problem

Existing MLLM serving systems (e.g., vLLM, SGLang) use a tightly coupled architecture that routes text-only and multimodal requests through the same pipeline and colocates all inference stages (preprocessing, encoding, prefill, decode) on the same GPUs. This causes resource contention between stages with vastly different compute demands, inflates TTFT under heavy multimodal load, and prevents per-stage parallelism tuning. Static decoupled systems (e.g., DistServe) address stage separation but fail under bursty multimodal workloads because resource allocation is fixed.

Method

ElasticMM organizes serving at two hierarchical levels:

Modality level — Modality-Aware Load Balancing: Requests are classified and routed to either a text-only group or a multimodal group. A proactive mechanism allocates idle GPU instances to maximize burst tolerance (ratio of peak-available to average-required instances per group, Eq. 1). A reactive mechanism handles sudden spikes by preempting instances from less-loaded groups using a gain-cost model that weighs acceleration gain against migration and performance-impact cost.

Stage level — Elastic Partition Scheduling: Within each modality group, the inference pipeline is disaggregated into encode, prefill, and decode stages. A stage-level manager performs: (1) FCFS request dispatching with memory and compute-bound tipping-point estimation; (2) stage allocation that preempts decode instances to expand prefill parallelism when gain exceeds migration cost (Eq. 2); (3) elastic auto-scaling that shrinks decode parallelism to a minimum and triggers reactive scaling when decode bottlenecks are detected (Eq. 3). Data Parallelism is preferred within a stage to avoid expensive weight migrations; tensor parallelism is used only when a single GPU cannot fit the model.

Inference optimizations: (1) Unified Multimodal Prefix Cache — a two-pool LRU cache covering both vision tokens (hash-keyed by raw image content) and text prefix KV states; cache hits skip re-encoding and/or re-prefilling. (2) Non-blocking Encoding — image preprocessing and encoding are isolated into an asynchronous process/instance, decoupling encoding latency from prefill scheduling.

Key Contributions

• Identification of tight coupling as the root bottleneck in MLLM serving and proposal of EMP, a two-tier (modality-level + stage-level) elastic serving paradigm.
• Modality-aware load balancing combining proactive greedy allocation and reactive gain-cost-driven instance preemption.
• Elastic partition scheduling with dynamic parallelism adjustment per inference stage using formalized gain-cost models.
• Unified multimodal prefix caching combining vision-token and text-prefix KV caches under a single LRU eviction policy.
• Non-blocking asynchronous encoding that eliminates head-of-line blocking between encode and prefill stages.

Results

Evaluated on 8× NVIDIA A800 80GB GPUs with LLaMA3.2-Vision-11B (encoder-decoder) and Qwen2.5-VL-7B (decoder-only) on ShareGPT-4o and VisualWebInstruct datasets; baselines are vLLM v0.6.6 and a multimodal-extended DistServe.

• TTFT vs. vLLM: up to 4.2× reduction (Qwen2.5-VL, ShareGPT-4o); 3.5× (LLaMA3.2-Vision, ShareGPT-4o); 3.7× and 2.9× on VisualWebInstruct.
• Throughput vs. vLLM: 3.2–4.5× higher throughput across SLO scales 1×–5× (ShareGPT-4o).
• Throughput vs. DistServe: up to 2.3× higher throughput.
• EMP ablation (vs. best static allocation): 1.8× throughput gain on Qwen2.5-VL; 2.3× on LLaMA3.2-Vision.
• Optimization ablation: Unified prefix cache alone significantly reduces TTFT; adding non-blocking encoding yields further consistent reduction; both are additive.

Limitations

• Evaluation restricted to a single-node 8-GPU testbed; multi-node performance under inter-node communication latency is explicitly left to future work.
• Gain-cost models for preemption decisions rely on offline-profiled thresholds and a tunable penalty factor w, which may require re-calibration across different hardware configurations.
• Reactive scaling during inter-group preemption incurs KV cache migration overhead that is amortized only when burst is sustained; transient spikes may temporarily degrade latency.
• No comparison against model-level optimizations (token pruning, KV pruning); the paper explicitly excludes accuracy-degrading methods, so quality-efficiency trade-offs are not explored.

Relevance to Vision-Language Models

ElasticMM directly addresses the deployment scalability of VLMs in production serving scenarios, where heterogeneous request streams (text vs. image-text) and long visual token sequences (6,500–7,400 tokens per 904×904 image) stress existing systems. The modality-group isolation design is particularly relevant for encoder-decoder VLMs (e.g., LLaMA3.2-Vision, Flamingo) where cross-attention prevents efficient mixed batching. The unified prefix cache is a practical efficiency gain for VLM deployments with repeated images or system prompts (e.g., multi-turn visual dialog). This work complements model-side VLM research by providing the serving infrastructure needed to deploy larger, higher-resolution VLMs efficiently.