Skip to content

LLM-Powered Fully Automated Chaos Engineering: Towards Enabling Anyone to Build Resilient Software Systems at Low Cost

🕒 Published (v1): 2025-11-11 06:03 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ChaosEater is an LLM-powered agentic system that automates the entire Chaos Engineering (CE) cycle for Kubernetes systems — from hypothesis formulation through fault injection, analysis, and system reconfiguration — at costs of \(0.21–\)0.84 and 11–25 minutes per cycle. It predefined a six-phase agentic workflow assigning subdivided CE processes to role-specific LLM agents. Case studies on NGINX and Sock Shop demonstrate consistent, qualitatively validated CE cycles with minimal human involvement.

Problem

Chaos Engineering requires multi-domain expertise and is largely manual: while fault execution and metric collection have been automated by tools like Chaos Mesh, k6, and Prometheus, hypothesis formulation, experiment planning, and system reconfiguration still require human engineers. This makes CE inaccessible, costly, and slow — especially as LLM-generated software proliferates and resilience validation becomes more urgent.

Method

ChaosEater wraps a predefined, sequential agentic workflow around the four-phase CE cycle, decomposed into six operational phases:

  1. Pre-processing: Deploy the K8s system via Skaffold; LLM agents summarize manifests, identify potential resilience issues, filter harmful prompts.
  2. Hypothesis: An LLM agent defines measurable steady states (e.g., pod count, p95 latency), inspects current values via auto-generated Python/JavaScript scripts using the K8s API or k6, and defines thresholds. The scripts are augmented with assertions — called Validation as Code (VaC) — to mechanically validate states. A separate agent defines a failure scenario and maps it to Chaos Mesh fault parameters (structured JSON, stepwise refinement).
  3. Experiment: An LLM agent plans a three-stage chaos experiment (pre-validation → fault injection → post-validation) and generates a Chaos Mesh workflow manifest. Execution is fully automated; ChaosEater waits passively.
  4. Analysis: VaC script pass/fail results are checked mechanically. On failure, an LLM agent analyzes logs, timeline, and manifests to produce a countermeasures report.
  5. Improvement: An LLM agent rewrites K8s manifests (gradually increasing redundancy) and triggers an improvement loop — repeating Experiment → Analysis → Improvement until all VaC scripts pass.
  6. Post-processing: Summarizes the completed CE cycle; outputs modified K8s manifests.

The underlying LLMs (GPT-4o-2024-08-06 at temperature 0) require no fine-tuning.

Key Contributions

  • First end-to-end LLM-based system automating the full CE cycle (hypothesis through system reconfiguration), not just partial phases.
  • Validation as Code (VaC): steady-state validation encoded as executable unit tests rather than LLM judgment, ensuring consistency and transparency.
  • Quantitative cost characterization: single CE cycles cost \(0.21–\)0.84 and 11–25 minutes across small and large-scale K8s systems.
  • Open-source release of all ChaosEater resources.
  • Qualitative validation by 2 human engineers and 3 LLMs (GPT-4o, Claude Sonnet 3.5, Gemini Pro 1.5), all scoring every phase ≥3/5.

Results

  • NGINX (2 manifests): $0.21, 11 min; correct reconfiguration in 5/5 runs; 59k input / 5.9k output tokens.
  • Sock Shop (29 manifests, ~10× scale): $0.84, 25 min; correct reconfiguration in 4/5 runs; 284k input / 13k output tokens.
  • Cost scales sub-linearly: >10× resource increase yields only ~4× cost increase.
  • All five runs per system completed without runtime errors (100% stability).
  • All evaluators (human and LLM) rated every CE phase above the positive threshold (≥3/5) for both systems.
  • Identified and fixed concrete resilience bugs: restartPolicy: Never in NGINX (replaced Pod with Deployment); single-replica front-end in Sock Shop (increased replicas).

Limitations

  • Deployable only in development environments due to security concerns about autonomous fault injection in production.
  • Improvement scope restricted to K8s manifest reconfiguration; cannot modify application code (Python), frontend (HTML/CSS/JS), or infrastructure-as-code (Terraform).
  • Single CE cycle only; no long-horizon vulnerability exploration across cycles or temporal system-state tracking.
  • Evaluation limited to two case studies with manually introduced resilience bugs — no broad benchmark or adversarial test suite.
  • No comparison against human engineer baselines with actual time/cost measurements.

Relevance to Harnesses / Meta-Harnesses

ChaosEater is a domain-specific meta-harness: it predefined a fixed agentic workflow that orchestrates multiple specialized LLM agents across sequenced phases, each with scoped tools, prompts, and outputs — precisely the architecture pattern of a harness. The improvement loop (Experiment → Analysis → Improvement until convergence) is a concrete example of a self-correcting harness loop driven by executable test results (VaC) rather than model self-evaluation. The VaC pattern — encoding validation criteria as runnable code that the harness executes to gate loop termination — is directly applicable to other harness designs where ground-truth signals are available. This work demonstrates that predefined, rigid agentic workflows (rather than fully dynamic planning) can be sufficient and cost-effective for well-structured multi-phase tasks.