Towards Scalable Customization and Deployment of Multi-Agent Systems for Enterprise Applications

🕒 Published (v1): 2026-06-16 21:30 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a production-ready framework from Capital One that distills a large teacher LLM into a compact 10B student model for an automotive-retail multi-agent chatbot, then stacks EAGLE speculative decoding and FP8 quantization to achieve a 4.48× throughput speedup over the original 70B teacher with no measurable quality loss. The two-stage pipeline covers both model customization (CPT → SFT → DPO) and inference optimization, targeting strict enterprise SLA requirements.

Problem

Deploying multi-agent LLM systems in production is bottlenecked by three compounding problems: (1) cumulative latency from multiple sequential LLM calls per request, (2) massive GPU memory footprint of large models, and (3) high per-token generation cost that prefill optimization alone cannot address. Existing work treats domain adaptation and inference optimization as independent concerns; no unified end-to-end framework addresses both simultaneously for agentic workflows.

Method

The framework has two stages:

Stage 1 — Agentic Model Customization distills a Llama 3 70B teacher (\(\pi_T\)) into a 10B student (\(\pi_\theta\)) via a four-step sequential pipeline: - Stage 0: Block expansion of Llama 3.1 8B (one new transformer block inserted after every four original blocks, zero-initialized so the model starts as an identity map). - Stage 1 (CPT): Context-aware Continual Pretraining on ~5T tokens; a sample-specific context prefix \(C_x\) is prepended to smooth per-token loss spikes at sequence boundaries, mitigating catastrophic forgetting. Model merging is applied after each stage. - Stage 2 (SFT): LoRA fine-tuning on judge-refined traces \(y^* = J(y, \Phi_\text{refine})\), where a judge model \(J \gg \pi_T\) rewrites teacher outputs. - Stage 3 (DPO): Preference alignment on (prompt, \(y^*\), \(y\)) triples to close the residual capability gap and correct teacher errors.

Training data is generated by an agentic user simulator \(S\) that conditions on conversation history \(H_{<i}\), intent definitions \(\mathcal{N}\), seed topics \(\mathcal{I}\), and environment context \(\mathcal{E}\).

Stage 2 — Inference Optimization stacks two complementary techniques on \(\pi_\theta^\text{BF16}\): - EAGLE speculative decoding: A 250M draft model trained on 127k samples (77k open-source + 50k proprietary synthetic); greedy decoding of the draft outperforms tree decoding at serving concurrency. - FP8 W8A8 post-training quantization: Static per-tensor min-max PTQ with mixed (public + in-domain) calibration data to control activation clip rates at long context lengths (≥1024 tokens).

Key Contributions

• End-to-end teacher→student distillation pipeline combining CPT, SFT (LoRA), and DPO with a judge-refinement loop and agentic user simulator for synthetic data generation.
• Context-aware CPT that prepends document-level context to reduce loss spikes and catastrophic forgetting during domain adaptation.
• Domain-specific EAGLE drafter trained on mixed proprietary + open-source dialogue traces; demonstrates that synthetic in-domain data (1.37×) outperforms external data alone (1.19×), and combining both reaches 1.78× speedup in greedy mode.
• Mixed calibration data strategy for FP8 PTQ, showing a 6.1× lower activation clip rate at 2048 tokens vs. public-only calibration—directly relevant to production prompts that routinely exceed 1000 tokens.
• Empirical evidence that DPO is essential: CPT alone yields near-total E2E failure (4.2% pass rate), SFT recovers most capability, and DPO closes the final gap to exceed the teacher.
• 4.48× end-to-end throughput improvement over the 70B teacher with no measurable quality degradation.

Results

• \(\pi_\theta^\text{EAGLE+FP8}\) achieves 4.48× QPS over \(\pi_T\) (70B teacher): 6.54 vs. 1.46 QPS; P90 latency 0.92 s vs. 3.92 s.
• \(\pi_\theta^\text{BF16}\) alone gives 2.33× speedup over teacher and exceeds teacher accuracy on Planner Agent and Understander Agent (Figure 3).
• \(\pi_\theta^\text{EAGLE+FP8}\) maintains 100% E2E stress-test pass rate (120 scenarios, 10 turns each), matching \(\pi_\theta^\text{BF16}\).
• EAGLE drafter ablation: external-only 1.19×, synthetic-only 1.37×, combined 1.46× (tree decoding); greedy + combined reaches 1.78×.
• FP8 mixed calibration reduces activation clip rate to 6.1× lower than public-only at 2048-token context length with 100% E2E pass rate vs. 97.27% (public-only) and 98.18% (in-domain-only).
• Distillation evaluated on 1,424 simulated conversations (8,848 data points); stage-by-stage CPT→SFT→DPO progression documented per agent.

Limitations

• Results are specific to a single automotive-retail enterprise domain; generalization of exact speedup numbers to other domains is not guaranteed.
• Distillation pipeline depends on a single model family acting as both teacher and judge—biases and domain blind spots in those models propagate to the student; manual DPO pair curation is required to patch business-logic edge cases.
• Generating and verifying hundreds of thousands of multi-turn synthetic traces requires substantial upfront compute.
• EAGLE drafter must be retrained whenever system prompts or business logic change, because distribution shifts degrade draft token acceptance rates.
• FP8 inference optimizations require NVIDIA Hopper GPUs (H100); fallback to higher precision eliminates the reported throughput gains, limiting hardware portability.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct industrial instantiation of a production multi-agent harness: a fixed sequential pipeline (Understander → Planner → Evaluator → Executor → Explainer) with a replanning feedback loop, governed by a meta-level inference and training orchestrator. The end-to-end pipeline—agentic simulator, judge-refinement loop, staged CPT/SFT/DPO distillation, and stacked inference optimization—is itself a meta-harness that automates the creation and deployment of the inner agent workflow. For researchers tracking harness design, the key insight is that harness-level throughput bottlenecks require coordinated optimization across data engineering, model adaptation, and serving infrastructure simultaneously, not in isolation. The study also quantifies how harness data quality (simulator fidelity, calibration data mixture) upper-bounds agent performance, a broadly applicable finding for anyone building automated agent training pipelines.