SwarmX: Agentic Scheduling for Low-Latency Agentic Systems

🕒 Published (v1): 2026-06-19 13:11 UTC · Source: Arxiv · link

Why this paper was selected

GPU-CPU scheduling for multi-call agentic pipelines; latency-critical harness infrastructure

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SwarmX is a cluster scheduling system for multi-model agentic applications that replaces heuristic routers/scalers with lightweight neural predictor agents. It reduces P99 latency by 44–52% in production and P95 latency by up to 61.5% on a controlled testbed by making scheduling decisions aware of prompt semantics, device state, and downstream model-call structure. The core insight is that both inference time and multi-hop model-call graphs are prompt-dependent, so scheduling requires distributional neural prediction rather than round-robin or average-cost heuristics.

Problem

Existing cluster schedulers (Ray, Kubernetes, Murakkab) use prompt-agnostic policies (round-robin, random, power-of-two) or simple statistical predictors (linear regression, random forests). These fail on agentic workloads because: (1) a single agent harness request can run seconds-to-minutes depending on prompt semantics, so poor routing persists and amplifies tail latency rather than being amortized at high QPS; and (2) downstream model invocations—their number, identity, and branching structure—are only revealed at runtime, making resource scaling reactive and error-prone.

Method

SwarmX reframes routers and scalers as scheduler agents that observe state and take bounded scheduling actions via existing infrastructure interfaces.

Predictor architecture. Each predictor decouples into (i) a small semantic model—an isomorphic smaller variant of the target LLM (e.g., 35M-param model for an 8B-param target)—that extracts prompt-level features, and (ii) a lightweight MLP that combines semantic features with device, runtime, and target-model features to output a distributional prediction (quantile sketches over latency or downstream call counts).

Distribution-aware scheduling. Rather than reducing predictions to point estimates, SwarmX maintains per-queue or per-replica demand state as quantile sketches \(\mathbf{Q}\). For each candidate action \(a\), it hypothetically composes the predicted distribution \(D_a\) into the affected sketch via quantile-grid composition \(\oplus\), evaluates a tail-cost functional \(C_{\text{tail}}(\mathbf{Q}(a))\), samples a probability-weighted candidate subset, and commits the selected action. This preserves tail uncertainty across sequential decisions.

Training. The semantic model is trained with a configurable per-sample loss \(\mathcal{L}_{\text{sem}}\) over prompt-to-response-property mappings (Eq. 1). Router/scaler MLPs use weighted pinball loss over prescribed quantile levels \(\tau_k\) (Eq. 2). All predictors are trained from execution logs.

Online adaptation. A sliding-window tail pinball loss monitors per-prompt/device-group prediction drift. When average loss exceeds threshold \(\theta\), only the MLP is asynchronously retrained (semantic model is stable unless the target model itself changes).

Scheduler-agent framework. Predictors, training data, adaptation logic, and action interfaces are wrapped in a common substrate that plugs into Ray, Kubernetes, or any existing cluster manager without replacing them.

Key Contributions

• Compact neural predictors jointly conditioning on prompt semantics (via isomorphic small variant), device characteristics, runtime state, and target-model properties—running on CPU or a small GPU fraction.
• Distribution-aware routing and scaling via quantile-sketch composition, enabling tail-cost optimization rather than mean-latency optimization.
• Unified online adaptation: MLP-only retraining for workload/device drift, full semantic-model retraining only on target-model changes.
• Scheduler-agent framework providing a plug-in interface to augment existing Ray/Kubernetes infrastructure.

Results

• Production cluster (~1,000 heterogeneous GPUs, millions of CPU cores): SwarmX sustains up to 2× throughput of prior production schedulers under the same SLO; reduces P99 latency by 44–52% across multi-agent code generation, deep research, and text-to-video workloads.
• Controlled 128-GPU testbed: reduces end-to-end P95 latency by up to 61.5% versus state-of-the-art schedulers (Ray, Murakkab, Power-of-Two Choices), including on open-ended workloads whose call structure is entirely runtime-determined.
• Baselines: Ray [round-robin/random], Kubernetes, Power-of-Two Choices, Murakkab (average per-model estimates), PSC/Cilantro (statistical predictors).

Limitations

• Predictor accuracy depends on the quality and coverage of production execution logs; cold-start or rare prompt distributions may yield poor initial predictions.
• The semantic model must be an isomorphic variant of each target model; heterogeneous or frequently updated model fleets increase maintenance overhead.
• Adaptation retrains only the MLP for workload drift—if prompt-to-response behavior shifts (e.g., new task types), the semantic model retraining trigger may lag.
• Evaluation covers three workload families (coding agent, deep research, text-to-video); generalization to other agentic topologies is asserted but not benchmarked.
• The paper is truncated before the full implementation and evaluation sections, so quantitative breakdown of overhead (prediction latency, MLP size impact) is partially obscured.

Relevance to Harnesses / Meta-Harnesses

SwarmX explicitly models the agent harness as a first-class scheduling primitive: it treats the multi-hop model-call graph that a harness orchestrates as a prompt-dependent structure to be predicted and provisioned for, not merely as a black-box workload. This is directly relevant to anyone building or studying meta-harnesses, because it shows that the scheduling substrate underneath a harness—how GPU/CPU replicas are provisioned across the harness's sequential and parallel model calls—is a critical and largely unsolved systems problem. The scheduler-agent framework design pattern (bounded agents augmenting existing infrastructure) mirrors the meta-harness pattern of wrapping existing components rather than replacing them, and the online adaptation mechanism is a practical blueprint for keeping harness-serving infrastructure aligned with evolving orchestration logic.