Skip to content

Securing the AI Agent: A Unified Framework for Multi-Layer Agent Red Teaming

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AI-Infra-Guard is an open-source red-teaming framework from Tencent Zhuque Lab that organizes AI agent security assessment around a "layer-paradigm matching" principle: four attack-surface layers (infrastructure, protocol/tool, agent behavior, model) each require a fundamentally different detection paradigm. The framework is the first open-source system to span all four layers within a single server-agent architecture, including supply-chain auditing of agent skill packages.

Problem

The rapid deployment of AI infrastructure (model-serving engines, agent platforms, MCP ecosystems) has outpaced available security tooling. Three gaps make conventional tools inadequate: (1) a cataloguing gap — mainstream fingerprinting databases lack entries for AI components; (2) a versioning gap — AI projects use irregular versioning (build IDs like b7824, 2.3.dev, rolling latest tags) that defeat semantic-version comparison; (3) a threat-model gap — the dominant AI risks (unauthenticated GPU access, credential leakage, prompt injection, tool poisoning, alignment erosion) differ from the injection/XSS flaws classical scanners target. No existing tool spans all layers of the AI stack.

Method

The framework operationalizes a layer-paradigm matching principle formalized through four evidence classes (signature, semantic, behavioral, statistical) and four corresponding detection modules:

  • M1 — Infra-Scan (infrastructure layer): A declarative fingerprinting and vulnerability-matching engine written in Go, with a hand-written expression interpreter and a custom version-normalization procedure. It maintains 75+ AI component fingerprints and 1,400+ vulnerability rules, stratified into verified, version-based, and inferred findings by confidence.

  • M2 — MCP-Scan (protocol/tool layer): An LLM-driven agentic harness that audits MCP server code and tool metadata for semantic flaws (tool poisoning, command injection, credential mishandling) that no fixed rule can express. It operates in both white-box static mode and black-box dynamic mode. Introduces "Prompt-as-Rule": vulnerability knowledge encoded as natural-language detection criteria, which the LLM applies as if they were rules. The scanner treats its own inputs as untrusted to mitigate indirect prompt injection against itself.

  • M3 — Agent-Scan (application/behavioral layer): A capability-aware black-box red-teaming pipeline that probes a deployed agent through multi-turn adversarial dialogue. Uses escalation ladders, stop rules, and canary-token verification to bound cost while covering four risk families (system-prompt disclosure, tool misuse, indirect prompt injection, privilege escalation).

  • M4 — Prompt-Security (model layer): A jailbreak evaluation harness integrating 26+ attack operators (single- and multi-turn) over 16 red-teaming datasets, with LLM-as-judge scoring of attack success rate (ASR) as the statistical measure of alignment robustness.

  • Agent Skills Scanning (Section 7, extending M2): Applies the same LLM-driven paradigm to auditing skill/plugin packages in the agent supply chain for poisoning, over-privilege, and embedded prompt injection, realizable either as a server-side harness or as a pure host-model specification.

The architecture uses a central web server that dispatches tasks to worker agents over WebSockets; Go handles the infrastructure module in-process, Python subprocesses handle LLM-driven modules.

Key Contributions

  • Formal articulation of the Security Heterogeneity Principle and four evidence classes (signature, semantic, behavioral, statistical) motivating layer-adaptive assessment.
  • M1: Declarative fingerprint/vulnerability language for AI software with non-standard versioning; precision stratification of findings.
  • M2: "Prompt-as-Rule" paradigm for LLM-audited MCP servers; adversarial posture in which the scanner mistrusts its own inputs.
  • M3: Cost-bounded, capability-aware black-box agent red teaming via escalation ladders and canary-token verification.
  • M4: Unified jailbreak harness with 26+ attack operators across 16 datasets and LLM-as-judge.
  • Supply-chain module (M2 extension): First open-source auditing of agent skill packages for poisoning and prompt injection.
  • Open-source release of rule corpora, Prompt-as-Rule criteria, attack operator library, and all detection modules.

Results

The paper is a system/technical report; concrete benchmark numbers are reported in later sections (partially truncated in the provided text). From what is available: - Infrastructure scanner covers 75+ AI components and 1,400+ vulnerability rules. - Jailbreak harness spans 26+ attack operators across 16 datasets. - Agent skill benchmarking results and ASR comparisons are described as present (Sections 7.6, 8.6, 9.5) but numerical values are not included in the provided excerpt.

Limitations

  • Benchmark numbers and head-to-head comparisons against prior tools are not fully visible in the provided text, making independent verification of claimed detection effectiveness impossible from this excerpt.
  • The Prompt-as-Rule paradigm depends on the auditor LLM's reliability; the paper acknowledges engineering effort is required for consistent structured outputs but does not quantify false-negative rates for the LLM-driven modules.
  • Black-box agent red teaming (M3) is inherently incomplete: coverage is bounded by cost, and adversarial dialogues can miss vulnerabilities that require rare multi-turn sequences.
  • Supply-chain skill auditing is static/semantic; runtime-only vulnerabilities in skills are not caught by M2/skill scanner and must be caught by M3.
  • The system is designed for red teaming/auditing, not real-time runtime defense; it does not block attacks, only surface findings.

Relevance to Agentic AI / LLM Agents

This paper directly addresses the security posture of deployed LLM agents, covering attack surfaces that are unique to agentic systems: MCP tool poisoning, tool shadowing, confused-deputy attacks, indirect prompt injection through retrieved content, and supply-chain compromise via agent skill packages. The "Prompt-as-Rule" paradigm is a methodologically novel contribution — encoding security policy as natural language that an LLM auditor evaluates — with implications for scalable, updatable security specifications as the MCP/tool ecosystem evolves rapidly. The four-layer decomposition (infrastructure → protocol → behavior → alignment) provides a useful conceptual taxonomy for anyone studying or building agentic AI systems, since each layer maps to a distinct failure mode that prior agent-security work has addressed in isolation. The open-source release makes this the most comprehensive public red-teaming baseline available for the AI agent stack.