Skip to content

Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents

🕒 Published (v1): 2026-05-28 13:27 UTC · Source: Arxiv · link

Why this paper was selected

Agora: LLM agents autonomously detect bugs in production-level consensus protocols

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Agora is a domain-aware three-agent LLM framework that autonomously detects protocol-level logic bugs in distributed consensus implementations by combining hypothesis-driven testing (HDT) with specialized agents for orchestration, attack-scenario synthesis, and test generation. Applied to four consensus protocol codebases (Raft, EPaxos, HotStuff, BullShark), Agora discovers 15 previously unknown zero-day logic bugs while all single-agent ReAct baselines find zero protocol-level bugs. The key insight is that detecting safety-violating logic bugs requires global state reasoning across execution stages, which single-agent approaches structurally cannot do.

Problem

Existing LLM-based bug detection tools (LLMSAN, SCALE, LLMDFA, PrimeVul, KNighter) excel at localized, implementation-level bugs (memory leaks, integer overflows, simple logic errors) but cannot reason about complex, state-dependent protocol behaviors spanning multiple components and execution stages. Consensus protocol bugs that violate safety (consistency) or liveness properties—the bugs with catastrophic consequences in production databases, storage systems, and blockchains—require understanding global invariants across distributed node interactions, which is beyond the reach of single-agent, code-reading-tool-augmented LLMs.

Method

Agora implements a three-agent pipeline grounded in the HDT formalism, where a vulnerability hypothesis is a tuple \(H = (C, A, E, O)\): preconditions \(C\), action sequence \(A\), expected buggy behavior \(E\), and oracle assertions \(O\).

Three specialized agents (each with four subtasks, 12 total steps per iteration): 1. Orchestrator: Maintains global state \(S\); runs bug-exploitation (uses previously found bugs to guide new hypotheses) and state-analyzer (prevents redundant scenario exploration); routes context to other agents via a hub-and-spoke communication topology. 2. Strategy: Runs constraints-analyzer (distinguishes CFT vs. BFT constraints—e.g., malicious node assumptions are valid in BFT but wasteful in CFT) and attack-scenario-generator (produces scenarios by controlling node join/crash/recover events and message ordering). 3. TestGen: Employs a reflection-loop (up to num_iter=50 retries) to generate, execute, and analyze unit tests against the target repository; outputs structured bug reports if a test exposes a logic violation.

Knowledge library: Bug patterns crawled from GitHub issues (filtered to protocol-level only) and protocol constraints are stored in lightweight system memory shared across all agents. Agents access codebases via MCP. The loop terminates on budget exhaustion (3-hour wall-clock limit per experiment). Temperature is set to 0.2 for all LLMs.

Key Contributions

  • First multi-agent system combining HDT with consensus-domain knowledge to target protocol-level logic bugs specifically.
  • Taxonomy of five protocol-level bug categories: Recovery & Execution Divergence, Persistence & Monotonicity Violations, Dependency & Topology Flaws, Message Binding & Signature Violations, Resource & Operational Visibility Violations.
  • Discovery of 15 zero-day protocol-level logic bugs across Raft (etcd), EPaxos, HotStuff, and BullShark (sui), with a false positive rate of 26.1% (34 real bugs out of 46 generated reports).
  • Ablation showing each component is critical: removing any single component reduces bug detection by 73–100%.

Results

  • Agora vs. baselines: Agora finds 15 logic bugs (1 Raft, 9 EPaxos, 4 HotStuff, 1 BullShark); all single-agent ReAct baselines (GPT-5.2, Gemini 3.0 Pro Preview, Claude Sonnet 4.5, Qwen3 Coder 480B) find 0 logic bugs.
  • Implementation bugs: Baselines collectively surface 22 implementation bugs (all spurious from the protocol-safety perspective); Agora surfaces 0, confirming it correctly focuses resources on logic bugs.
  • Ablation (using a single unspecified LLM on the same 4 repos):
  • Full Agora: 11 bugs
  • w/o bug-exploitation: 3 bugs (−73%)
  • w/o state-analyzer: 0 bugs (−100%)
  • w/o constraints-analyzer: 1 bug (−91%)
  • w/o scenario-generator: 0 bugs (−100%)
  • w/o reflection-loop: 0 bugs (−100%)
  • False positive rate: 26.1% across all 46 TestGen reports from all four LLMs.
  • Bug distribution: Recovery & Execution Divergence and Persistence & Monotonicity Violations each account for 4 bugs (26.7% each); Dependency & Topology Flaws: 3; Message Binding & Signature Violations: 2; Resource & Operational Visibility Violations: 2.

Limitations

  • Evaluated on only four consensus protocols; generalization to other distributed protocol families (e.g., two-phase commit, Viewstamped Replication) is untested.
  • 3-hour compute budget per experiment is a significant resource constraint; scalability to larger or more complex codebases is not characterized.
  • False positive rate of 26.1% still requires manual triage; the reflection loop does not fully eliminate spurious reports.
  • Bug pattern library and constraints require manual curation from GitHub issues and protocol documentation; keeping them current as protocols evolve is a maintenance burden.
  • Communication topology (hub-and-spoke through Orchestrator) concentrates bottlenecks; the paper does not analyze Orchestrator failure modes or context-window saturation on long runs.
  • Experiments use temperature 0.2 with a fixed LLM suite; sensitivity to temperature or model choice beyond the four tested is not explored.

Relevance to Harnesses / Meta-Harnesses

Agora is a concrete instantiation of a domain-specialized multi-agent meta-harness: it wraps four distinct LLM backends behind a fixed three-agent orchestration topology, manages shared state (knowledge library, global state \(S\)) across agents, and implements a bounded reflection loop—precisely the architectural patterns central to harness design. The hub-and-spoke communication contract (Orchestrator as state manager, Strategy and TestGen as stateless workers) is a reusable harness pattern applicable to any complex verification domain requiring role specialization and iterative refinement. The ablation results quantifying per-component contribution (each removal causes 73–100% performance drop) provide unusually rigorous evidence that harness component design is load-bearing, not merely organizational—a key empirical result for anyone designing meta-harnesses where modularity and state management are active research questions. The use of MCP for codebase access also situates Agora within the emerging standard for tool-equipped agent harnesses.