Skip to content

DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems

🕒 Published (v1): 2025-12-07 09:23 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DoVer (Do-then-Verify) is an intervention-driven debugging framework for LLM multi-agent systems that validates failure hypotheses by replaying execution traces with targeted edits rather than relying solely on log-based attribution. It segments failed session logs into trials, proposes failure hypotheses per trial, generates concrete interventions (plan edits or instruction adjustments), and re-executes from the intervention point to measure outcome improvement. On GAIA and AssistantBench with MagneticOne, DoVer recovers 18–28% of failed trials; on GSMPlus with AG2, it recovers 49%.

Problem

Log-only failure attribution for LLM multi-agent systems produces untested hypotheses and is ill-posed for two reasons: (i) attribution accuracy is capped by noisy ground-truth labels—annotator uncertainty of 15–30% with ~20% initial disagreement—because sessions contain multiple planning–execution trials each with independent failure points; (ii) inter-agent misalignment makes single-agent/single-step attribution inherently ambiguous (e.g., orchestrator issues bad instruction, sub-agent executes wrong action). State-of-the-art LLMs achieve below 10% step-level attribution accuracy on the WW benchmark under the standard protocol.

Method

DoVer operates as a four-stage pipeline over a session log \(\tau = \{(a_t, m_t, \sigma_t)\}_{t=1}^{T}\):

  1. Trial segmentation: an LLM identifies re-plan steps to split \(\tau\) into trials \(\tau^i\), shortening context per causal chain and enabling parallel hypothesis testing.
  2. Failure attribution: for each trial \(\tau^i\), generate hypothesis \(h_i = (\hat{a}^i_{\hat{t}}, r^i_{\hat{t}})\) (suspected agent, step index, rationale) using a prompt-based All-at-Once approach over the trial log.
  3. Intervention generation: convert \(h_i\) into a concrete edit—either (a) modified instructions from the orchestrator to sub-agents or (b) a revised orchestrator plan—applied at the attributed step.
  4. Intervention execution: replay from the intervention point forward, preserving all prior steps, yielding counterfactual trace \(\tilde{\tau}_I = \{\tau_1, \ldots, \tau_{i-1}, \tilde{\tau}_i\}\). Outcomes are scored via (a) Trial Success Rate and (b) Progress Made \(= \frac{A(\tilde{\tau}_I) - A(\tau)}{K}\), where \(A(\cdot)\) counts LLM-verified milestones against human-annotated solution steps (\(K \le 5\)). Each intervention is run three times to reduce LLM stochasticity.

Hypothesis validation classifies each trial into Validated (\(\ge 2/3\) runs succeed), Partially Validated (\(<2/3\) succeed but \(\ge 2/3\) fulfill the intervention and achieve \(>20\%\) progress), Refuted (intervention fulfilled, no progress), or Inconclusive (intervention not followed by agent).

Key Contributions

  • DoVer framework: the first intervention-driven automated debugging pipeline for LLM multi-agent systems operating across heterogeneous agent frameworks.
  • Empirical analysis of ground-truth annotation uncertainty: 14 of 29 GAIA cases in the WW benchmark have uncertain labels; step attribution accuracy rises from 44% to ~24% when moving from certain to uncertain cases.
  • Two prompt refinements (step indices + annotator-guidance reminders) that raise GPT-4o attribution accuracy from 6% to 24% on WW-HC.
  • Outcome-oriented evaluation metrics (flip rate, milestone-based progress, hypothesis validation categories) that sidestep brittle attribution accuracy.
  • Generality demonstrated across two agent frameworks (MagneticOne, AG2) and three datasets (GAIA, AssistantBench, GSMPlus).

Results

  • MagneticOne + AssistantBench: 18% of failed trials flipped to success.
  • MagneticOne + GAIA: 28% of failed trials flipped to success; up to 16% milestone progress.
  • AG2 + GSMPlus: 49% of failed trials recovered.
  • Hypothesis validation: 30–60% of hypotheses validated or refuted (range depends on task complexity); remaining cases are Inconclusive (agent fails to follow the intervention).

Limitations

  • Interventions restricted to orchestrator level (message edits, plan updates); cannot directly improve sub-agent capabilities (e.g., extending WebSurfer with in-page search), requiring system-level modifications beyond the framework's scope.
  • Milestone-based progress metric requires human-annotated intermediate solution steps; inapplicable when these are unavailable (LLM-as-judge alternative left for future work).
  • Inconclusive category is unavoidable: agents may not faithfully execute the intervened instruction, making it impossible to distinguish a bad hypothesis from a sub-agent compliance failure.
  • Only synchronous agent executions are supported; asynchronous multi-agent logs are out of scope.
  • Three re-runs per intervention multiplies compute cost linearly.

Relevance to Harnesses / Meta-Harnesses

DoVer is structurally a meta-harness: it wraps around existing agent frameworks (MagneticOne, AG2) to intercept, segment, edit, and replay their execution traces without modifying the underlying agent logic. Its trial-segmentation and replay-with-state-restore infrastructure—preserving \(\sigma_t\) (browsing history, LLM context) up to the intervention point—is precisely the checkpoint/replay machinery that a general multi-agent meta-harness must expose. The framework's prompt-based trial segmentation (rather than system-specific log patterns) demonstrates a design principle for harnesses that must generalize across heterogeneous agent backends. For researchers building harnesses, DoVer's intervention taxonomy (orchestrator-plan edits vs. sub-agent instruction adjustments) and its counterfactual trace evaluation protocol are directly applicable to runtime error recovery and automated test-repair loops within a meta-harness.