Skip to content

Scalable Inference Architectures for Compound AI Systems: A Production Deployment Study

🕒 Published (v1): 2026-04-28 14:53 UTC · Source: Arxiv · link

Why this paper was selected

Production deployment study of compound AI systems; practical inference architectures

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This production deployment study from Salesforce describes a modular, serverless inference architecture built to serve compound AI systems (multi-model agent pipelines) at enterprise scale. The system achieves >50% P95 latency reduction, up to 3.9× throughput gain, and 30–40% cost savings over static GPU deployments. It introduces rigorous measurement of compound-system-specific failure modes—fan-out overhead, cascading cold starts, and heterogeneous scaling dynamics—that are absent from single-model serving literature.

Problem

Traditional ML serving infrastructure (static GPU endpoints, uniform autoscaling) was designed for single-model workloads and breaks down when serving compound AI systems where one user request fans out to 3–5 heterogeneous model invocations with complex dependency graphs. Specific failure modes include: (1) fixed 24/7 GPU costs with no per-component scaling; (2) resource contention across model types sharing a GPU pool; (3) cascading cold-start latency where dependent models cannot warm in parallel; (4) deployment pipelines too coarse-grained for multi-component iteration cycles.

Method

The architecture decouples orchestration from inference serving via a tiered design:

  • Orchestration layer: The Atlas Reasoning Engine runs a directed, event-driven graph of function nodes (retrieval, planning, action execution) over a publish-subscribe async pattern, parallelizing independent branches.
  • Prediction Service (gateway): A unified inference gateway with per-model routing tables, a request priority queue (fast-path vs. slow-path models), latency-class metadata tagging, and per-component circuit breakers for structured failure handling.
  • Inference backend: Serverless-first (scale-to-zero) with spill-over to dedicated instances via capacity-threshold triggers in "auto" mode; each model endpoint scales independently.
  • Compound-aware scaling: Per-model invocation rate tracking rather than aggregate user-request scaling; three-tier cold-start mitigation: (a) coordinated pre-warming of downstream dependencies when any pipeline member is first accessed, (b) tiered provisioned concurrency only for the critical-path (LLM) model, (c) predictive scheduled warm-up from traffic signals.
  • MLOps pipeline (Falcon): YAML-configured CI/CD; model version switching without code change; component-level A/B testing via independent per-model routing.

Key Contributions

  • First production characterization of compound-system-specific inference challenges: fan-out amplification (\(N\) user requests → up to \(5N\) model calls), cascading cold starts (effective latency = serial dependency chain, not max individual cold start), and asymmetric heterogeneous scaling dynamics.
  • Coordinated pre-warming strategy that reduces compound cold-start latency by 65% (from ~180s to ~65s) by parallelizing warm-ups across dependency chains.
  • Hybrid deployment routing (serverless + dedicated per-model) reducing total inference cost by an additional 15–20% versus pure-mode strategies.
  • Quantitative analysis at production scale: 136B tokens/month, 2,250 RPS peak capacity across 21 global inference regions, 8.7Ă— year-over-year request growth.
  • Component-level A/B testing capability reducing model experimentation cycles from weeks to hours.

Results

  • P95 latency: >50% reduction overall; ApexGuru (13B) at high concurrency: ~37s (legacy SageMaker) → ~10–11s (new system); >57% P95 reduction across Atlas Reasoning Engine tool calls.
  • Throughput: Legacy single-GPU saturated at 50–60 RPM; autoscaling system sustained 200+ RPM (~3.9Ă— gain).
  • Cost: 30–40% reduction for ApexGuru (eliminating overnight idle GPU); mixed-mode deployment saved additional 15–20% vs. either pure strategy.
  • Fan-out overhead: 45–80ms coordination tax at warm state (<2% of 5–8s agent response time); vs. 1.5–3s serial overhead in legacy architecture.
  • Cold start: Coordinated warming reduced compound cold-start latency 65% (180s → 65s); P95 SLA breach rate dropped from ~2% to <0.5% with coordinated warming.
  • Scaling accuracy: SQL executor correctly scaled 2–3Ă— under 10Ă— traffic spike; uniform scaling would have over-provisioned it 3–5Ă—.
  • Reliability: >95% agent availability during partial model outages; <0.05% message drop rate at peak load.
  • Quality: Atlas Reasoning Engine achieved 2Ă— response relevance increase and 33% higher end-to-end task success vs. single-LLM; ~33% more support cases resolved without human intervention.

Limitations

  • Cost analysis is informal; no rigorous model comparing serverless per-token pricing vs. dedicated GPU break-even points across workload shapes.
  • Coordinated pre-warming requires explicit pipeline registration; unregistered or dynamically composed pipelines cannot benefit from dependency-aware warm-up.
  • Predictive warming relies on coarse-grained traffic predictability (business hours, batch schedules); unpredictable burst traffic (e.g., viral events) still triggers cascading cold starts.
  • Vendor-neutral portability claimed but only "validated in prototypes" on Nvidia NIM/private Kubernetes; production multi-cloud is not demonstrated.
  • Quality improvement figures (2Ă— relevance, 33% task success) lack methodology detail—no description of evaluation set, metric definitions, or statistical significance.
  • The study is from a single organization's production system; generalizability to different compound topologies or model counts is unverified.

Relevance to Harnesses / Meta-Harnesses

This paper is directly relevant to anyone building meta-harnesses: it addresses precisely the infrastructure layer that sits beneath a harness—how to reliably invoke, scale, and recover multiple heterogeneous model backends when a harness fans out to many concurrent subagents or tools. The compound-aware scaling primitives (per-model rate tracking, dependency-graph-aware pre-warming, hybrid serverless/dedicated routing) represent the production-grade execution substrate that a research harness assumes but rarely specifies. The cascading cold-start problem and circuit-breaker patterns are especially important for meta-harnesses that run long, multi-phase pipelines where a stalled component in phase \(k\) can block all of phase \(k+1\). Operational lessons—YAML-driven model swapping, component-level A/B testing, per-model circuit breakers—map directly onto desiderata for configurable, modular research harnesses that need to iterate individual pipeline stages independently.