Skip to content

MTTR-A: Measuring Cognitive Recovery Latency in Multi-Agent Systems

🕒 Published (v1): 2025-11-08 21:29 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MTTR-A introduces a runtime reliability metric that quantifies cognitive recovery latency in LLM-based multi-agent systems (MAS), adapting classical dependability theory (MTTR, MTBF, availability) to the orchestration layer where failures are semantic rather than infrastructural. The framework defines complementary metrics (MedTTR-A, MTBF, NRR) with formal theorems bounding steady-state cognitive uptime, validated on a LangGraph benchmark across 200 runs.

Problem

Existing runtime monitoring tools for LLM-based MAS describe failures but do not quantify how quickly distributed reasoning recovers once coherence is lost. Classical MTTR/MTBF metrics apply to binary up/down infrastructure faults; they cannot capture cognitive faults where agents remain functionally running while reasoning drifts, memories desynchronize, or coordination degrades. No prior work defines a quantitative, orchestration-level metric for cognitive recovery latency.

Method

The framework models the MAS as a system \(S = \{A_1, A_2, \ldots, A_N\}\), each agent \(A_i\) experiencing fault-recovery episodes with detection timestamp \(t_{f,i}^{(j)}\) and restoration timestamp \(t_{r,i}^{(j)}\). Per-agent recovery duration is \(\Delta T_i^{(j)} = t_{r,i}^{(j)} - t_{f,i}^{(j)}\), decomposed as:

\[\Delta T^{(j)} = T_{\text{detect}} + T_{\text{decide}} + T_{\text{execute}}\]

System-level MTTR-A averages over \(N\) agents:

\[\text{MTTR-A}_{\text{sys}} = \frac{1}{N}\sum_{i=1}^{N} \text{MTTR-A}_i\]

The Normalized Recovery Ratio (NRR) is \(\text{NRR}_{\text{sys}} = 1 - \lambda_{\text{sys}}\mu_{\text{sys}}\), proven to be a conservative lower bound on steady-state cognitive uptime \(\pi_{\text{up,sys}}\) under an alternating-renewal model (Theorem 1). A confidence-aware extension \(\text{NRR}_\alpha = 1 - \lambda_{\text{sys}}(\mu + k_\alpha \sigma)\) incorporates recovery-time variance (Theorem 2). Five reflex families (recovery, human-in-the-loop, runtime control, coordination, safety) define the control vocabulary. Empirical evaluation uses a LangGraph three-node workflow (reasoning → drift-check → recovery) on the AG News corpus with a cosine-similarity drift threshold \(\tau_{\text{drift}} = 0.6\) across 200 runs.

Key Contributions

  • Adaptation of MTTR, MTBF, and availability ratios to cognitive orchestration, defining MTTR-A as a first-class runtime metric for reasoning recovery in MAS.
  • A five-category reflex taxonomy (recovery, human-in-the-loop, runtime control, coordination, safety) providing a standardized vocabulary for cognitive fault management.
  • Formal theorems establishing NRR as a conservative lower bound on steady-state cognitive uptime (Theorem 1) and a variance-aware confidence-graded extension NRR\(_\alpha\) (Theorem 2).
  • An open LangGraph benchmark (200 runs, AG News corpus) measuring MTTR-A, MTBF, and NRR across four reflex modes with structured JSONL telemetry.

Results

  • Overall MedTTR-A across 200 runs: \(6.21\,\text{s} \pm 2.14\,\text{s}\); 90th-percentile recovery latency: \(10.73\,\text{s}\).
  • System MTBF: \(6.73\,\text{s} \pm 2.14\,\text{s}\) (text truncated before per-reflex breakdown).
  • Measurable, distinct recovery latencies observed across all four reflex modes (auto-replan, rollback, tool-retry, human-approve), confirming the metric's discriminative power.

Limitations

  • Benchmark is a simulation with synthetic drift injection (stochastic perturbations + cosine threshold), not production agent workloads; real semantic drift may have a different distribution.
  • Drift detection is proxied by retrieval confidence score, not a general semantic coherence measure — does not generalize to non-retrieval tasks without re-instrumentation.
  • MTTR-A is agnostic to reasoning quality; a system can score low MTTR-A while producing incorrect outputs after recovery.
  • System-level MTTR-A\(_{\text{sys}}\) is a simple average across agents, which may mask heterogeneous per-agent recovery dynamics.
  • The paper is truncated before complete per-reflex quantitative results are reported.
  • Evaluation uses only LangGraph; generalization to other orchestration frameworks (AutoGen, CrewAI, etc.) is asserted but not demonstrated.

Relevance to Harnesses / Meta-Harnesses

MTTR-A directly addresses a gap in meta-harness design: harnesses that orchestrate multi-agent pipelines need not only to detect failures but to measure and minimize recovery latency at the orchestration layer. The proposed Cognitive Reliability Layer — with Open Cognitive Telemetry, Causal Drift Graphs, policy hooks, and audit interfaces — maps closely to the instrumentation and control plane components that a production meta-harness would need to implement. The reflex taxonomy (rollback, replan, sandbox-execute, human-approve) provides a concrete action vocabulary for harness-level fault handling. For researchers building or evaluating meta-harnesses, MTTR-A offers the first quantitative runtime SLO (e.g., "median recovery < 7s, NRR > 0.9") against which harness designs can be benchmarked.