OpenRCA 2.0: From Outcome Labels to Causal Process Supervision

🕒 Published (v1): 2026-06-25 15:24 UTC · Source: Arxiv · link

Why this paper was selected

OpenRCA 2.0 adds causal process supervision labels; stronger eval signal for multi-step reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing RCA benchmarks for LLM agents only label the root-cause component, not the causal propagation path, enabling agents to score well by pattern-matching while producing unverifiable reasoning. OpenRCA 2.0 introduces PAVE, a forward-verification annotation protocol that reconstructs causal paths from fault-injection records, yielding 500 step-wise annotated instances. Across 11 frontier LLMs, exact root-cause recovery succeeds in only 20.7% of cases, and ~14.5% of "correct" identifications lack any grounded propagation path.

Problem

All major RCA benchmarks evaluate agents solely on the outcome (did the model name the right component?) and ignore how the agent arrived there. This allows ungrounded diagnoses: an agent identifies the correct root-cause service but without a verified causal propagation path connecting it to the observed symptom. Outcome-only scoring hides this failure mode because constructing ground-truth paths from telemetry alone is as hard as the agent's task — both face the same ill-posed backward inference.

Method

PAVE (Path Annotation via Verified Effects) exploits the information asymmetry of fault-injection experiments: annotators know the intervention \(\text{do}(v_{\text{root}})\), turning backward inference into forward verification.

Given the dependency graph \(G=(V,E)\), raw telemetry \(O(t)\), and the recorded intervention, PAVE runs two phases:

1. Structural Pruning — projects telemetry onto a discrete state alphabet \(\Sigma\) via a node-type-aware mapping \(\Psi\), then applies a propagation rule set \(R\) (encoding known RPC-timeout cascades, resource-contention patterns, etc.) over a time-expanded heterogeneous graph with nodes \((v, s, t)\); a constrained DFS yields a candidate path set \(\Pi_\text{cand}\).

2. Causal Verification — each candidate path is verified against a pre-injection reference baseline \(D_\text{base}\) (approximating \(P(\cdot \mid \text{do}(v_\text{root}=\emptyset))\)) using distributional tests (Z-score for Gaussian metrics, percentile-based for heavy-tailed). A path \(\pi\) is admitted to \(\Pi^*\) only when: (i) every node along \(\pi\) shows statistically significant deviation, (ii) every edge conforms to \(R\), and (iii) downstream anomaly onset is strictly after upstream onset — all three conditions jointly.

Agents follow a tool-augmented ReAct-style architecture (Deep Research style), returning structured output: root-cause claims and propagation edges each paired to a re-executable SQL query as evidence.

Key Contributions

• PAVE protocol: first principled method to reconstruct step-wise causal ground truth from fault-injection recordings via forward verification
• OpenRCA 2.0: 500-instance cross-system benchmark spanning 3 microservice architectures (TrainTicket 44 services, OTel Demo 15 services, Hotel Reservation 9 services) and 27 fault types; 7.5 verified causal edges per instance on average
• Ungrounded diagnosis concept: formalises the gap between AnySvc (correct service named) and Path Reachability (correct service grounded in a verified causal graph path)
• Process-level metrics: Node F1, Edge F1, and Path Reachability alongside outcome metrics (EM, F1, AnySvc)
• Failure mode taxonomy: three patterns — premature commitment, presence bias on observable activity, salience capture on loudest signal

Results

• Exact Match (EM): 20.7% mean across 11 models; best is Gemini 3.1 Pro at 29.4%
• AnySvc (at least one correct service named): 76.0% mean — surface-level appearance of competence
• Path Reachability (PR): 61.5% mean — ~14.5 pp gap from AnySvc constitutes ungrounded diagnoses
• Node F1 (mean 62.2%) vs. Edge F1 (mean 43.4%): 18.8 pp gap holds for every model; Gemini shows widest split (67.9% Node F1, 36.8% Edge F1)
• Outcome F1: 34.1% mean; Gemini 3.1 Pro leads at 43.8%
• No single model dominates all columns: per-column maxima distributed across four different models
• PR/AnySvc conditional rate discriminates models hidden by outcome scores: Qwen3.6-Max ≈91% vs. Claude Sonnet 4.6 ≈66% despite comparable F1
• Annotation reliability: 94/100 case-level inter-annotator agreement; all 6 disagreements were under-coverage (missing weakly-supported edges), zero spurious-edge disagreements

Limitations

• PAVE requires fault-injection infrastructure with known interventions; cannot be applied to production incidents where the root cause is not known a priori
• The conjunctive three-condition gate is deliberately strict, preferring precision over recall in path annotation — some valid propagation edges may be excluded
• 500 instances across three systems may not cover the full diversity of real-world microservice architectures or fault types
• Agent evaluation uses identical tool suites and prompts, isolating backbone reasoning but not reflecting real operational agent diversity
• Silent injections (faults that produce no observable downstream signature) are filtered out, potentially underrepresenting stealthy fault types

Relevance to Agentic AI / LLM Agents

RCA in microservice systems is a canonical stress test for agentic capabilities — long-context telemetry ingestion, multi-step tool use, and abductive causal reasoning — making this benchmark directly relevant to anyone evaluating or improving LLM agents on complex reasoning tasks. The core finding that outcome-only evaluation dramatically overstates agent competence (76% AnySvc vs. 61.5% Path Reachability) is a methodological warning applicable to any agentic benchmark that scores final answers without auditing the reasoning chain. The PAVE protocol operationalises process supervision as a scalable annotation strategy using information asymmetry from controlled experiments, a technique transferable to other agentic evaluation domains where ground-truth derivation paths are hard to label. The failure taxonomy (premature commitment, presence bias, salience capture) gives concrete targets for improving LLM agent reasoning in observation-rich, multi-hop diagnostic settings.