Skip to content

MIRROR: Novelty-Constrained Memory-Guided MCTS Red-Teaming for Agentic RAG

🕒 Published (v1): 2026-06-25 09:26 UTC · Source: Arxiv · link

Why this paper was selected

MCTS red-teaming for multimodal agentic RAG; novel attack taxonomy across orchestrator layers

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MIRROR is a unified red-teaming framework for multimodal agentic RAG systems that combines memory-guided Monte Carlo Tree Search (MCTS) with a deterministic Novelty Gate to generate attacks across four heterogeneous surfaces (text poisoning, image poisoning, direct query, orchestrator manipulation). It addresses a critical failure mode in prior automated red-teaming: high measured ASR that is almost entirely benchmark replay rather than novel attack discovery. MIRROR achieves 76% ASR on image poisoning (vs. 52% baseline) and 97% ASR on orchestrator attacks at half the query cost, with the lowest cross-surface variance of all evaluated methods.

Problem

Existing automated red-teaming methods for RAG systems are surface-specific and recycle known attacks from fixed seed pools: on text-poisoning benchmarks, PAIR/TAP/PS exhibit 73–84% exact duplication, so reported ASR overstates genuine vulnerability discovery. No prior framework spans all four agentic-RAG attack surfaces (text, image, direct query, orchestrator), and evaluation protocols conflate in-loop simulator success with real-target verification, inflating apparent effectiveness.

Method

MIRROR operates via two coupled phases:

  1. Episodic memory bank: Seeded from the released ART-SAFE-BENCH corpus (41,815 records), each entry stores attack traces with taxonomy metadata. A Prior Network indexes entries via ChromaDB (cosine similarity, text-embedding-3-small, 1536-dim), retrieving top-\(k\) memories conditioned on (goal \(g\), surface \(b\)) to induce operator priors and a per-case rejection set.

  2. Novelty-constrained MCTS planning: Search uses PUCT-style selection: $\(a^*(s) = \arg\max_a \left[ Q(s,a) + c_{\text{puct}} \cdot P(a|s) \cdot \frac{\sqrt{N(s)}}{1 + N(s,a)} \right]\)$ where \(P(a|s)\) is derived from retrieved memory strategy tags via \(P(a|s) \propto \epsilon + \sum_{j: a \in \Gamma(\text{tags}(m_j))} \exp(\beta \sigma_j)\). The mutation action space \(\mathcal{O}\) includes rewriting, Unicode/homoglyph obfuscation, encoding transforms (base64, ROT13, hex), payload injection, persona shifts, multi-turn decomposition, and visual steganography (B2).

  3. Novelty Gate: A deterministic hard constraint rejects any candidate \(x\) where \(\text{norm}_{\text{ex}}(x) \in \mathcal{B}_b^{\text{ex}}\) or \(\text{norm}_{\text{alnum}}(x) \in \mathcal{B}_b^{\text{alnum}}\), where \(\mathcal{B}_b\) is the union of the benchmark pool, the per-case retrieved neighbor set, and the within-session accepted set. Novelty is enforced before any simulator or target query.

  4. Two-stage verification: Simulator (gpt-5-mini) rollouts guide exploration; final success requires deterministic replay on the real target within budget \(Q\).

Key Contributions

  • MIRROR framework: Unified MCTS planner across all four agentic-RAG attack surfaces with retrieval-derived operator priors and a hard deterministic Novelty Gate preventing benchmark replay.
  • ART-SAFE-BENCH v2.0.0: 41,815 in-package records (+ runtime adapters for 41,991+ total) spanning B1–B4, with SHA-256 integrity manifest, deterministic case IDs, and taxonomy/provenance metadata.
  • Reproducible verification protocol: Mandatory deterministic target replay, Wilson 95% CIs, query-efficiency reporting, and patched-knownset stress tests; Novel-ASR and DupBench@Exact metrics as complements to raw ASR.

Results

  • Cross-surface consistency: MIRROR achieves CV = 0.47 across surfaces; all baselines collapse—GCG: CV = 1.38 (79% B1, 1% B3); TAP: CV = 1.41 (72% B1, 0% B3).
  • B2 image poisoning: MIRROR 76% vs. Text Overlay 52%, LSB 32%.
  • B4 orchestrator attacks: MIRROR 97% ASR at query efficiency Q/Success = 1.00 vs. Toolflip 86% at 2.08.
  • B3 direct query: MIRROR 31% ASR (highest among all evaluated methods).
  • B1 novelty: PAIR/TAP/PS achieve 58–77% raw ASR but 6–9% Novel-ASR after deduplication (73–84% DupBench@Exact); MIRROR achieves 0% DupBench@Exact by construction, so Novel-ASR = ASR = 47%.
  • Patched-knownset stress test: At \(K_{\text{known}} = 10{,}000\), PAIR/TAP reach 93–97% SelfDup@Exact (self-collapse); MIRROR maintains low duplication.
  • Domain shift (CYBER RAG SOC target): Baselines outperform MIRROR, indicating corpus-target alignment and simulator fidelity are binding variables for retrieval-derived priors.

Limitations

  • Novelty Gate is exact-match only under two normalizations (\(\text{norm}_{\text{ex}}\), \(\text{norm}_{\text{alnum}}\)); semantically equivalent paraphrases pass undetected; embedding- or entailment-based novelty filters are deferred due to threshold sensitivity.
  • B2 image duplication metrics are reported as "–" since text-based normalization does not apply to image carriers without a dedicated visual similarity instrument.
  • On B1, specialized suffix search (GCG) achieves higher raw ASR (79% vs. 47%); MIRROR trades surface-specialist performance for cross-surface breadth with novelty certification.
  • No strict ablation isolating contributions of memory priors, Novelty Gate, and verification individually; remains future work.
  • Domain shift degrades performance: on the structured-output CYBER RAG target (strict JSON schema), baselines outperform MIRROR, exposing sensitivity to corpus-target distribution alignment.
  • Attacker-side API costs (mutator, simulator, judge calls) are orthogonal to the victim-query budget axis and not reported in main results.

Relevance to Agentic AI / LLM Agents

MIRROR directly targets the security of agentic RAG architectures—systems where an LLM orchestrator autonomously sequences retrieval, tool calls, and reasoning—representing a realistic deployment class increasingly used in enterprise settings. The paper's treatment of orchestrator-level attacks (B4: tool-flip and argument manipulation via schema-valid but unauthorized decisions) is particularly relevant to the agentic AI safety literature, as it isolates the tool-selection boundary as a distinct vulnerability surface beyond prompt injection. The novelty-constrained MCTS framing offers a principled methodology for systematic vulnerability discovery that avoids the benchmark-gaming failure mode common in automated red-teaming. The ART-SAFE-BENCH corpus and the recommendation to report Novel-ASR alongside raw ASR provide concrete infrastructure and evaluation norms for the growing subfield of agentic system security evaluation.