Aegis: Automated Error Generation and Attribution for Multi-Agent Systems

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; automated error generation and attribution essential for debugging MAS

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Aegis is an automated framework that converts successful multi-agent system (MAS) executions into labeled failure trajectories by injecting context-aware errors via an LLM-based manipulator. It produces 9,533 annotated trajectories across six MAS frameworks and six task domains, addressing the severe data scarcity that blocks progress on MAS error attribution. Fine-tuned small/medium LLMs trained on Aegis data match or exceed proprietary models an order of magnitude larger on error attribution benchmarks.

Problem

Error attribution in LLM-based MAS—identifying which agent caused a failure and what error mode it committed—is fundamentally data-starved. Existing benchmarks (Who&When: 184 errors; MASFT: ~150 tasks; TRAIL: 148 traces) are tiny and rely on costly expert annotation. Meanwhile, SOTA LLMs show limited out-of-the-box attribution ability, creating a scalability deadlock: large datasets are needed but prohibitively expensive to label manually.

Method

Aegis follows a three-stage pipeline:

1. Collect deterministic baselines. Run each MAS (temperature=0, fixed seed) to obtain correct trajectories \(\tau_\text{corr}\) where the outcome evaluator \(Z(\tau_\text{corr}) = 0\).

2. Inject errors via an adaptive manipulator \(\mathcal{M}_\text{manip}\). For each baseline, the manipulator generates multiple injection plans \(P_\text{inj}^{(j)} = \{(n^*, Y^*), \ldots\}\) specifying target agents and error modes from a 14-mode MAST taxonomy (spanning Specification Issues, Inter-Agent Misalignment, Task Verification Failures). The manipulator applies either Prompt Injection (corrupting an agent's input context) or Response Corruption (replacing the agent's output) at temperature 0.7 to ensure diversity. Errors are task-relevant and context-aware (e.g., infinite loops in code, plausible wrong values in math).

3. Validate and label. Only trajectories where the injected fault causes system failure (\(Z(\tau_\text{inj}) = 1\)) are retained. Ground-truth attribution \(G(\tau_\text{inj}) = P_\text{inj}^{(j)}\) is known by construction—no human annotation required.

The resulting dataset supports three learning paradigms: - SFT: Minimize negative log-likelihood \(\mathcal{L}_\text{SFT}(\theta) = -\sum \log p_\theta(o|x)\) where \(o\) is a JSON attribution map. - RL (GRPO): A hierarchical reward \(R\) awards cpair for exact (agent, error) pairs, partial credit cagent/cerror for correct agent or error mode alone, penalizes false positives and duplicates, and normalizes by \(S_\text{max} = |P_\text{gt}| \cdot c_\text{pair} + c_\text{bonus}\). - Contrastive (DCL—Disentangled Contrastive Learning): Uses Multiple Instance Learning attention over trajectory turns, dual prototype banks \(B_A\) (agents) and \(B_E\) (error modes), and a composite loss \(\mathcal{L}_\text{DCL} = \lambda_\text{cls}\mathcal{L}_\text{cls} + \lambda_\text{con}\mathcal{L}_\text{con} + \lambda_\text{hier}\mathcal{L}_\text{hier}\), where \(\mathcal{L}_\text{hier}\) enforces the logical constraint that a predicted pair's probability cannot exceed the minimum of its constituent agent and error probabilities.

Key Contributions

• A reproducible, fully automated pipeline for converting successful MAS trajectories into labeled error trajectories, eliminating the manual-annotation bottleneck.
• Aegis dataset: 9,533 annotated trajectories across six MAS frameworks (MacNet, DyLAN, Debate, AgentVerse, MagenticOne, SmolAgents) and six benchmarks (MATH, GSM8K, HumanEval, SciBench, MMLU-Pro, GAIA).
• Three learning method implementations (SFT, GRPO-based RL, DCL contrastive) with a standardized multi-level evaluation protocol (Pair/Agent/Error × Micro-F1/Macro-F1).
• Open-source release of all code, data, and models.

Results

All scores are Micro-F1 / Macro-F1 (%) at the Pair level unless noted; full breakdown across six metrics reported.

• Qwen2.5-14B + SFT (Aegis-SFT): Pair µF1 16.62 → achieves Agent µF1 76.53 on Aegis-Bench, substantially outperforming baseline 14B without fine-tuning (5.47 Pair µF1, 35.78 Agent µF1).
• Qwen2.5-14B + GRPO (Aegis-GRPO): Pair µF1 6.84, Agent µF1 49.74 on Aegis-Bench; outperforms GPT-4.1 (7.44 Pair µF1, 37.48 Agent µF1) and approaches o3 (7.86, 40.31) despite being ~10× smaller.
• OOD generalization (Who&When): Qwen2.5-14B + SFT achieves 51.14 Error µF1 vs. 49.88 for the base model; Gemini-2.5-Flash leads large-scale models at 55.56 Error µF1.
• DCL (small-scale encoder model): Pair µF1 8.33, avg. score 12.61—substantially above the random baseline (0.33 Pair µF1) but below fine-tuned generative LLMs.
• Across all medium-scale models, SFT consistently yields the largest gains on Agent-level metrics; GRPO improves Pair-level precision.

Limitations

• All baseline executions use GPT-4o-mini at temperature 0, so injected error distributions may not reflect failures arising from more capable or stochastic agents.
• The 14-mode MAST taxonomy was derived from ~150 naturally-occurring traces; injected errors are constrained to this taxonomy and may miss novel real-world failure modes.
• Ground-truth labeling is by construction (injected plan = label), which conflates the attribution task with the injection plan rather than capturing emergent cascading failures that arise naturally.
• DCL contrastive approach significantly underperforms fine-tuned generative LLMs, suggesting the contrastive formulation for this task needs further development.
• OOD Who&When scores required LLM-based label mapping (Gemini-2.5-Flash) to convert free-text mistake reasons to MAST modes, introducing preprocessing noise.

Relevance to Agentic AI / LLM Agents

Debugging and root-cause analysis is one of the central unsolved reliability challenges for production multi-agent systems—a cascade failure in a 10-agent pipeline is nearly impossible to attribute manually at scale. Aegis directly attacks this by providing a scalable, automated data engine grounded in realistic error taxonomies, enabling smaller open-source models to approach proprietary model performance on attribution. The hierarchical RL reward design is particularly relevant to agent reliability research: it shows how graded, multi-level credit assignment (agent identification vs. error mode identification) can extract far more training signal from each failure trace than a binary success/failure signal. This work also demonstrates a general pattern—automated mutation of correct agent trajectories to produce labeled faults—that could generalize to other agent evaluation and self-repair problems beyond error attribution.