Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines

🕒 Published (v1): 2026-04-16 16:15 UTC · Source: Arxiv · link

Why this paper was selected

Serving agentic workflows via aggregate LLM pipelines; latency-throughput infrastructure

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Scepsy is a serving system that schedules multi-LLM agentic workflows onto GPU clusters by exploiting the observation that each LLM's fraction of total execution time is stable across requests even when absolute latencies vary wildly. It constructs an Aggregate LLM Pipeline abstraction to jointly optimize GPU allocation (replicas, tensor parallelism, fractional GPU shares) across all LLMs without requiring a fixed programming model. On a 16-GPU cluster, Scepsy achieves up to 2.4× higher throughput and 27× lower latency vs. baselines.

Problem

Existing agentic workflow serving systems either (a) optimize LLMs independently and ignore cross-LLM dependencies, or (b) require workflows to be expressed in a proprietary programming model and still leave GPU allocation to the user. Neither approach handles the combination of: arbitrary agentic frameworks (LangChain, AutoGen, Camel, LangGraph), unpredictable data-dependent control flow (branches, fan-out, loops), multi-LLM heterogeneity, and oversubscribed GPUs in small-to-medium self-managed clusters.

Method

Scepsy operates in four stages:

1. Workflow Tracing — an HTTP proxy sits in front of each LLM engine's completions API and records per-LLM request/response contents, timestamps, and a workflow ID, without touching the application code.
2. Statistical Aggregation — for each LLM \(m\), Scepsy extracts average invocations per workflow request \(n_m\) and average request-level parallelism \(p_m\) from overlapping timestamps. Dynamic behaviors (loops, fan-out, branches) are folded into these two scalars; non-LLM tool calls are discarded.
3. Aggregate LLM Pipeline construction — the workflow is abstracted as a latency/throughput pipeline over LLM stages, ordered by aggregate invocation fractions. Sequential and parallel calls are collapsed by adjusting per-stage latency contributions. The pipeline predicts end-to-end throughput (bottleneck stage) and latency (sum of adjusted per-stage latencies) for any candidate allocation.
4. Joint GPU Allocation + Placement — a scheduler searches over three dimensions per LLM: (i) replica count (data parallelism), (ii) tensor-parallelism degree, (iii) fractional GPU share. A hierarchical heuristic prunes the search space using performance statistics, hardware topology, and symmetries. Placement is topology-aware (NVLink-domain-respecting), enforced via a modified Kubernetes device plugin + Nvidia MPS for inter-fraction isolation.

Key Contributions

• Aggregate LLM Pipeline — a lightweight pipeline abstraction derived from execution traces; decouples allocation reasoning from agentic control-flow complexity.
• Framework-agnostic tracing — HTTP proxy captures LLM invocations without modifying the workflow code or requiring a specific API; compatible with LangChain, AutoGen, Camel, LangGraph, vLLM, etc.
• Joint three-dimensional GPU allocation — simultaneous optimization over replicas, tensor-parallelism degree, and fractional GPU shares; treats GPUs as quasi-continuous resources to avoid over-provisioning.
• Topology-aware fractional placement — NP-hard placement heuristic that respects NVLink domains, minimizes cluster fragmentation, and enforces isolation via Kubernetes + MPS; first such support layered on top of Kubernetes.
• Empirical stability insight — formal characterization that relative LLM execution-time fractions are up to 4× more stable than absolute latencies across beam-search requests.

Results

• Up to 27× lower average workflow latency vs. Kubernetes [18] and Aegaeon [56] at equal throughput (beam search workload, 16-GPU cluster).
• Up to 2.4× higher peak throughput vs. Kubernetes and Aegaeon at equal GPU budget.
• Evaluated on three workloads: RAG with reranking, beam search, and a combined workload.
• Beam search trace: per-request generator LLM invocations range 24–844; end-to-end latency ranges 9–264 s — yet relative execution-time fractions remain stable.

Limitations

• Profiling phase requires a representative set of workflow-level requests upfront; cold-start or distribution-shift scenarios are not addressed.
• The Aggregate LLM Pipeline discards non-LLM tool/orchestration time entirely; workflows with heavyweight tool calls (e.g., database scans, long web fetches) may see prediction error.
• Placement heuristic is approximate (NP-hard problem); optimality gap is not quantified.
• Stage ordering in the Aggregate LLM Pipeline is stated not to matter, which may lose accuracy for workflows with strong ordering-dependent latency profiles.
• Evaluation is limited to a 16-GPU cluster; scalability to larger heterogeneous clusters is not demonstrated.
• Prototype is 25K lines of Python; performance overhead of the HTTP proxy and MPS enforcement is not reported in the excerpt.

Relevance to Harnesses / Meta-Harnesses

Scepsy is directly relevant as infrastructure for running multi-LLM meta-harnesses at scale: a harness that orchestrates many LLM agents (e.g., a fan-out research workflow, a judge panel, or a pipeline of digest + trend agents) faces exactly the GPU oversubscription and unpredictable fan-out problems Scepsy solves. The framework-agnostic tracing layer is particularly important — it means a harness built on any orchestration library can be profiled and optimally scheduled without refactoring. For researchers building meta-harnesses that must run reliably and cheaply on self-managed clusters, the joint allocation approach offers a path to avoiding the manual GPU tuning that currently limits harness scalability.