When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systems

🕒 Published (v1): 2026-07-07 21:12 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; activation-based MAS security from Lei Ma's group; novel detection paradigm

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AcMAS is an activation-space framework for detecting malicious agents in LLM-based Multi-Agent Systems (MAS) without relying on explicit interaction graphs or output-level semantics. It detects compromised agents by measuring cosine divergence of internal LLM hidden states from a learned normal prototype, then steers corrupted activations back toward that prototype to restore functionality rather than isolating agents. It substantially outperforms graph-based baselines on both synchronous and asynchronous MAS under stealthy attacks.

Problem

Existing MAS security defenses rest on two fragile assumptions: (1) attacks are semantically explicit and detectable at the output level, and (2) agent interactions are globally synchronized so communication topology can be modeled as a graph (e.g., GNN). Modern attacks such as MINJA achieve semantic camouflage—benign-looking messages that corrupt internal reasoning while passing surface-level filters. Real-world MAS are increasingly asynchronous (event-driven, task-queue-based), causing round-aligned GNN inference graphs to diverge catastrophically from training graphs. Additionally, isolation-based mitigation (removing suspected agents) causes task collapse in tightly coupled systems.

Method

AcMAS operates in two stages:

1. Activation-based anomaly detection. For each agent \(A_i\) generating response at time \(t\), the hidden state of the final layer at the final generated token is extracted as \(h_i^{(t)} = \phi(M_i(x_i^{(t)})) \in \mathbb{R}^d\). A normal prototype \(\mu_\text{normal}\) is computed as the unit-normalized centroid of benign execution traces. Per-agent divergence is measured via cosine distance: $\(\delta_i^{(t)} = 1 - \left\langle \frac{h_i^{(t)}}{\|h_i^{(t)}\|},\, \mu_\text{normal} \right\rangle\)$ Agent \(A_i\) is flagged as compromised when \(\delta_i^{(t)} > \tau_\text{detect}\) (calibrated on a validation set; \(\tau_\text{detect}=0.12\)). For synchronous MAS the prototype is adaptively updated via momentum (\(\alpha=0.7\)) using agents not flagged in prior rounds. For asynchronous MAS a fixed prototype is maintained from initial benign traces, avoiding any need for temporal alignment.

2. Restorative latent intervention (activation steering). Instead of isolating detected agents, AcMAS injects a corrected activation: $\(\tilde{h}_i^{(t)} = h_i^{(t)} + \lambda_i \cdot (\mu_\text{normal} - h_i^{(t)})\)$ with adaptive correction strength \(\lambda_i = \min\!\left(1,\max\!\left(0, \frac{\delta_i^{(t)}-\tau_\text{detect}}{\tau_\text{max}-\tau_\text{detect}}\right)\right)\), where \(\tau_\text{max}=0.35\) captures the 95th percentile of malicious divergence. The corrected \(\tilde{h}_i^{(t)}\) is injected at the final layer during response regeneration.

Key Contributions

• Identifies three fundamental failures of existing MAS defenses: semantic camouflage, asynchronous incompatibility, and disruptive mitigation.
• Proposes topology-agnostic, output-agnostic malicious agent detection purely from LLM activation space, effective under both synchronous and asynchronous execution.
• Derives the activation clustering property: compromised agents exhibit systematically smaller pairwise cosine distances than benign agents, even when surface outputs appear benign.
• Proposes restorative latent intervention via activation steering as an alternative to agent isolation, preserving collaborative task structure.
• Demonstrates generalization across 4 LLM backbones (GPT-OSS-20B, DeepSeek-V3, LLaMA3-8B, Qwen3-30B-A3B), MAS scales of 8–80 agents, and attacker rates of 0%–40%.

Results

All results vs. best graph-based baselines (G-Safeguard / BlindGuard) on gpt-oss-20b backbone:

• Detection F1 (synchronous), averaged across 5 datasets: AcMAS 0.94–0.95 vs. G-Safeguard 0.70–0.74; improvement of +0.22 F1.
• Detection F1 (asynchronous): AcMAS 0.92–0.94 vs. G-Safeguard 0.41–0.46; improvement of +0.55 F1.
• Stealthy Prompt Injection (S-PI) on CSQA, sync: AcMAS F1=0.95 vs. G-Safeguard 0.74; AUROC 99.41 vs. 94.32.
• S-PI on CSQA, async: AcMAS F1=0.94 vs. G-Safeguard 0.46; FPR 0.05 vs. 0.45.
• Task completion rate (mitigation): 0.97 for AcMAS vs. 0.83 for isolation-based baselines (+0.14), with attack success rate reduced to 0.03.
• Architecture generalization: F1 consistently 0.92–0.97 across all four LLM backbones.
• TAM and PERM (non-graph, semantic-level baselines) perform substantially worse (F1 0.17–0.64 across settings).

Limitations

• Requires white-box access to agent LLM internals (hidden states); inapplicable to API-only black-box agents.
• Normal prototype initialization requires a clean set of benign execution traces; contamination of this reference set could degrade detection.
• Momentum-based prototype update (\(\alpha=0.7\)) may slowly absorb false negatives in high-attack-rate scenarios; robustness at extreme compromise fractions (>40%) is not fully characterized.
• Evaluated only on open-source LLM backbones; applicability to proprietary closed models is undemonstrated.
• Activation steering preserves role semantics approximately; heavily corrupted agents may produce residual malicious behavior post-correction that is not quantified per-attack-type.
• The paper does not evaluate adaptive adversaries who craft attacks specifically to minimize activation divergence from the normal prototype.

Relevance to Agentic AI / LLM Agents

Security of multi-agent LLM systems is a direct and growing concern as agentic pipelines are deployed in production: a single compromised sub-agent can cascade malicious behavior across an entire workflow. AcMAS introduces the key insight that internal activation patterns are more reliable attack signals than observable outputs, which is broadly relevant for any agentic monitoring or introspection system. The asynchronous-MAS robustness addresses a practical gap that topology-based approaches (GNN-based safeguards) cannot fill in event-driven orchestration frameworks like AutoGen or CrewAI. The restorative intervention paradigm—steering rather than isolating—represents a meaningful shift in how agent-level failures should be handled to preserve system-level task completion.