Skip to content

KRCA: An Efficient Root Cause Analysis System in Hyper-Scale Microservice Systems via Agentic AI

🕒 Published (v1): 2026-07-02 07:01 UTC · Source: Arxiv · link

Why this paper was selected

Production-scale agentic RCA system in hyper-scale microservices; industrial deployment evidence

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KRCA is a production-deployed, end-to-end root cause analysis system for hyper-scale microservice environments (200,000+ services) that combines API-level graph pruning, skeleton-constrained causal graph construction, and a memory-augmented multi-agent LLM framework to localize failing services and classify failure types within real-time budgets. It achieves AC@1 of 0.88 / 0.79 and reduces mean diagnosis time by 77.3% at Kuaishou. The core insight is that structured causal priors must gate LLM reasoning to prevent hallucination at scale.

Problem

Existing deep-learning RCA methods require frequent retraining whose cycle exceeds the system's variation cycle, making them impractical for highly dynamic hyper-scale systems. LLM-based RCA methods suffer from the "Lost in the Middle" dilemma: with hundreds of thousands of services each emitting 400+ metrics, causal evidence is buried in prohibitively long contexts, causing hallucinated diagnoses. Traditional causal discovery algorithms (PC, Granger) degrade sharply in accuracy (below 20%) and become computationally prohibitive when anomalous metric counts exceed ~15.

Method

KRCA is a three-stage pipeline:

  1. API-level drilldown: Starting from the alerting API, a recursive traversal of the dependency graph scores each downstream API with \(\text{Score}(P,C) = \max(\text{Score}_f(P,C), \text{Score}_l(P,C))\). The failure rate score is a time-lagged Pearson correlation over a dynamic window anchored at the failure inflection point; the latency score is a weighted composite of anomaly degree \(\mathcal{A}\), fluctuation contribution \(\mathcal{F}\), and correlation \(\mathcal{C}\). Paths below a threshold are pruned, yielding top-\(N\) suspicious services.

  2. Skeleton-based causal graph instantiation: Anomalous metrics (detected by Isolation Forest + PatternMatcher) are classified into four meta-metric types — External (E), Internal (I), Dependency (D), KPI (K) — and mapped onto a fixed generic causal skeleton \(G_s\) with predefined directional edges (e.g., E→I, E→K, I→D, D→K). This produces an initial intra-service causal graph \(G_o\) that constrains the downstream LLM reasoning to a sparse, high-recall structural prior rather than a free-form search.

  3. Multi-agent collaboration: A Main Agent coordinates parallel Sub Agents (each specializing in a cross-domain resource type, e.g., CPU, MySQL) to verify causal edges in \(G_o\) via tool calls. RAG supplies similar historical incidents and diagnostic experience through a tiered memory store. After iterative refinement and self-reflection, the final causal graph is used to localize the root cause service and classify the failure type, producing a structured failure report.

Key Contributions

  • First end-to-end RCA system demonstrated on a hyper-scale (200,000+ service) production environment, without requiring pre-trained or periodically retrained models.
  • API-level drilldown strategy that uses both failure-rate and composite latency scores to prune the dependency graph, reducing the candidate set far more effectively than service-level granularity.
  • Skeleton-based causal graph instantiation that replaces slow, inaccurate statistical causal discovery with a semantics-driven, expert-knowledge skeleton, providing structured constraints that keep LLM accuracy above 60% even at 20+ anomalous metrics.
  • Memory-augmented multi-agent reasoning framework with parallel Sub Agent execution, RAG over historical cases, and tiered memory, enabling real-time inference without sacrificing accuracy.
  • Production validation over six months on 300 real incidents at Kuaishou.

Results

  • AC@1 of 0.88 for root cause service localization; outperforms strongest baseline by +31 pp absolute.
  • AC@1 of 0.79 for failure type classification; outperforms strongest baseline by +32 pp absolute.
  • Average root cause localization time reduced by 77.3% in production vs. prior SRE manual process.
  • Empirical ablation: unconstrained LLM accuracy drops to ~30% at 20 anomalous metrics; skeleton-constrained LLM stays above 60% at the same scale.
  • Service-level dependency involves more than 10 services in 56.7% of alerts vs. 36.3% at API-level, validating the drilldown design choice.

Limitations

  • The causal skeleton \(G_s\) encodes a fixed four-type meta-metric taxonomy derived from Kuaishou's architecture; generalization to different middleware stacks or non-RPC service meshes requires manual skeleton re-specification.
  • Threshold parameters for the drilldown scoring function and the Sub Agent specialization boundaries are tuned for Kuaishou's environment and may not transfer directly.
  • The paper is truncated before fully describing the multi-agent collaboration module details (Section 3.4) and the complete ablation study design.
  • Evaluation uses 300 incidents from a single company; external generalization to other hyper-scale systems is unverified.
  • No discussion of adversarial or cascading failure scenarios where the skeleton prior may be systematically violated.

Relevance to Harnesses / Meta-Harnesses

KRCA is architecturally a production meta-harness: a Main Agent orchestrates multiple specialized Sub Agents in parallel, each scoped to a diagnostic subtask, with RAG and tiered memory providing shared context infrastructure — precisely the agent-orchestration pattern studied in harness research. The skeleton-constrained causal graph functions as a structured task decomposition layer, directing which causal edges each Sub Agent reasons about, analogous to how meta-harnesses decompose a problem into sub-prompts before dispatching specialized agents. The paper provides concrete evidence that imposing structured priors at the harness level (skeleton constraints) is essential to prevent LLM hallucination under scale, a general finding applicable to any multi-agent harness operating over large, noisy input spaces. Its production deployment at Kuaishou offers rare empirical grounding for the real-world latency and accuracy tradeoffs of agentic orchestration pipelines.