Multi-Agent LLM Orchestration Achieves Deterministic, High-Quality Decision Support for Incident Response

🕒 Published (v1): 2025-11-19 06:06 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A controlled 348-trial study using a containerized framework (MyAntFarm.ai) demonstrates that decomposing incident-response LLM inference into three sequential specialized agents (diagnosis → remediation → risk assessment) eliminates the output quality variance endemic to single-agent approaches, achieving 100% actionable recommendation rate versus 1.7% for a single-agent copilot. The primary value of multi-agent orchestration is deterministic quality, not latency: both architectures achieve ~40 s end-to-end. The study uses TinyLlama 1B on a single incident scenario, so generalization is explicitly flagged as future work.

Problem

Single-agent LLMs deployed in AIOps pipelines produce vague, non-executable incident-response recommendations (e.g., "investigate recent changes") 98.3% of the time. Existing LLM evaluation metrics (BLEU, ROUGE, BERTScore) measure linguistic similarity, not operational actionability, leaving a gap between detection and executable remediation that extends Mean Time to Resolution (MTTR).

Method

MyAntFarm.ai is a Docker Compose framework with five microservices: an Ollama/TinyLlama (1B, 4-bit quantized) backend, a single-agent FastAPI copilot (C2), a multi-agent coordinator (C3), an evaluator, and an analyzer. All 348 trials use a single fixed incident scenario (auth-service regression post-deployment).

C2 (Single-Agent): One LLM call with a multi-objective prompt requesting root cause, remediation commands, and risk assessment simultaneously (~200+ tokens).

C3 (Multi-Agent): Three sequential LLM calls, each with a focused single-objective prompt (~50–100 tokens): Agent 1 (diagnosis specialist) → Agent 2 (remediation planner, receives Agent 1 output) → Agent 3 (risk assessor, receives Agent 2 output). A non-LLM coordinator aggregates outputs. All three agents share the same TinyLlama backend; the architectural difference is prompt decomposition and sequential composition, not model diversity.

A novel metric, Decision Quality (DQ), is introduced: $\(DQ_i = \alpha V_i + \beta S_i + \gamma R_i, \quad \alpha=0.40,\ \beta=0.30,\ \gamma=0.30\)$ where \(V_i\) (validity) is the fraction of technically feasible actions, \(S_i\) (specificity) is scored via regex for version numbers and concrete commands, and \(R_i\) (correctness) is token overlap with a fixed ground-truth string. Recommendations with \(DQ > 0.5\) are classified as actionable. Statistical validation uses one-way ANOVA, Bonferroni-corrected pairwise \(t\)-tests, and Cohen's \(d\).

Key Contributions

• Empirical demonstration that multi-agent orchestration (sequential prompt decomposition, same model) yields 100% actionable rate vs. 1.7% for single-agent on 348 controlled trials.
• 80× specificity improvement and 140× correctness improvement over single-agent baseline.
• Zero quality variance in C3 (\(\sigma_{DQ} = 0.000\)) vs. C2 (\(\sigma_{DQ} = 0.023\)), enabling SLA commitments.
• Introduction of the Decision Quality (DQ) metric framework capturing validity, specificity, and correctness — operationally oriented rather than linguistically.
• Open-source reproducible framework (MyAntFarm.ai) with fixed seed, temperature, and Docker configurations.

Results

• T2U latency: C1 baseline 120.4 s, C2 41.6 s, C3 40.3 s — C2/C3 differ by only 3.2%.
• Mean DQ: C1 = 0.000, C2 = 0.403 ± 0.023, C3 = 0.692 ± 0.000.
• Actionability: C3 116/116 (100%) trials with DQ > 0.5; C2 2/115 (1.7%).
• Component breakdown: Validity identical (1.0 both); Specificity C2 = 0.007 vs. C3 = 0.557 (80×); Correctness C2 = 0.003 vs. C3 = 0.417 (140×).
• ANOVA: \(F(2, 345) = 18472.3\), \(p < 0.001\); pairwise C2 vs. C3: \(t(230) = -137.2\), Cohen's \(d = 18.2\).
• One C2 catastrophic outlier (4009 s, suspected LLM deadlock); no such failures in C3.

Limitations

• Single scenario: All 348 trials use one identical authentication-service incident; generalization across incident classes (network partitions, memory leaks, etc.) is untested.
• Simulated C1 baseline: Manual dashboard timing is Gaussian-jittered from practitioner literature estimates, not empirically measured.
• Minimal model: TinyLlama 1B is used for reproducibility; it is unclear how much of the C2/C3 gap shrinks with more capable models.
• Automated DQ scoring: Specificity uses regex; correctness uses token overlap against a short fixed ground-truth string — both are syntactic proxies that can be gamed and do not validate semantic correctness. Human inter-rater validation is deferred to Phase 2.
• Prompt engineering confound: Single-agent prompt is inherently harder (multi-objective); it is unclear how much of the advantage is architectural versus prompt quality.
• No memory, tool use, or iterative refinement: "Agents" are single inference calls — far simpler than autonomous agent frameworks; results may not generalize to richer agent architectures.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of the meta-harness pattern: a non-LLM coordinator orchestrates a fixed, sequential pipeline of specialized LLM inference calls, each scoped to a single subtask, and aggregates their outputs into a structured artifact. The key finding — that prompt decomposition and sequential composition alone (no model diversity, no memory, no tool use) produce deterministic, high-quality outputs from a weak 1B model — directly validates the core design principle of task-specialized sub-harnesses over monolithic prompts. The introduction of DQ as an operationally grounded evaluation metric for harness outputs (rather than linguistic similarity) is methodologically relevant: harness designers need metrics that measure whether the pipeline's output is executable, not just coherent. The zero-variance result further establishes that harness structure itself is a lever for determinism, independent of model stochasticity.