A Safety and Security Framework for Real-World Agentic Systems¶

🕒 Published (v1): 2025-11-27 00:19 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents a dynamic safety and security framework for enterprise-grade agentic AI systems, treating system-level risk as a composition of component-level risks that can cascade and amplify. It introduces an agentic risk taxonomy, an AI-driven red-teaming methodology (AIRA), and demonstrates end-to-end evaluation on NVIDIA's AI-Q Research Assistant, releasing a 10,796-trace dataset of attack/defense executions.

Problem¶

Traditional safety/security frameworks (e.g., CVSS) are insufficient for agentic systems because: (1) risks are compositional—component-level vulnerabilities (e.g., prompt injection via a tool) compound into system-level user harms; (2) non-deterministic LLM decision-making makes static pre-release testing blind to emergent, context-dependent risks; and (3) the separation between "safety" and "security" collapses in agentic settings, where adversarial security failures directly cause user harms.

Method¶

The framework operates in two phases embedded within the agentic workflow:

Phase 1 — Risk Discovery and Evaluation: Sandboxed AI-driven red teaming via the AIRA (Agentic Injection and Risk Assessment) blueprint. Attacker agents generate adversarial probes (injection probes and evaluation probes) against "threat snapshots"—captured intermediate states of the target agent's execution. Attack generation is automated and adaptive; probes are instrumented via OpenTelemetry (OTel) traces for full observability.

Phase 2 — Embedded Risk Mitigation and Continuous Monitoring: Defender agents apply layered, contextual defenses including prompt hardening and guard model interventions. Defenses are embedded close to risk sources within the live workflow rather than applied as perimeter controls, enabling early containment of cascading effects.

Risk is modeled compositionally: \(R_{\text{system}} = f(R_1, R_2, \ldots, R_n)\) where component risks \(R_i\) interact with compounding and second-order cascading effects. Risk prioritization uses a three-tier taxonomy (low/medium/high impact) across dimensions of autonomy level, blast radius, deployment prevalence, and remediation cost.

Key Contributions¶

Operational risk taxonomy for agentic systems covering tool misuse, cascading action chains, control amplification, memory poisoning, agent collusion, and goal misalignment—organized by discoverability, measurability, and criticality
Compositional risk assessment model that accounts for second-order interactions and compounding effects across LLMs, orchestrators, tools, memory, and data sources
AIRA red-teaming blueprint with automated attacker/evaluator agents using injection and evaluation probes against threat snapshots of the target workflow
Defender agent framework with layered defenses (prompt hardening + guard models) instrumented via OTel traces
Nemotron-AIQ Agentic Safety Dataset 1.0: 10,796 OTel trace files covering 2,596/2,600 security traces (without/with defense) and 2,200/2,200 content-safety traces (without/with defense), plus 400 adversarial and 200 benign runs per split

Results¶

Dataset contains traces of over 10,000 realistic attack and defense executions on the AI-Q Research Assistant workflow
Security subset: 2,596 traces without defenses, 2,600 traces with defenses (attack success rates measured under both conditions)
Content safety subset: 2,200 traces each for undefended and defended configurations
Guardrails (prompt hardening + guard model interventions) demonstrably reduce attack success rates; quantitative deltas are detailed in Sections 5–6 but not quoted in the excerpted text
Attack propagation patterns identified across agentic workflow steps via OTel span analysis

Limitations¶

Framework is demonstrated on a single target system (NVIDIA AI-Q Research Assistant); generalizability to other agentic architectures is asserted but not empirically validated across diverse systems
Risk taxonomy coverage and decision rules for category membership are derived from enterprise readiness criteria, which may not transfer to research or open-source deployment contexts
Static pre-release testing is critiqued, but the runtime overhead of continuous embedded monitoring is not quantified
Highly autonomous agents with unbounded action spaces remain hard to fully enumerate; the framework acknowledges non-determinism limits consistent risk evaluation
Dataset traces are specific to NVIDIA Blueprint's tool/orchestration stack; attack patterns may not generalize across other orchestrators

Relevance to Harnesses / Meta-Harnesses¶

This paper is directly relevant to harness design: the AIRA blueprint functions as a meta-harness—an orchestrating layer of attacker and defender agents that wraps a target agentic workflow, instruments it via OTel traces, injects probes at captured intermediate states, and evaluates outcomes without modifying the system under test. The "threat snapshot" mechanism (capturing intermediate execution state as a harness checkpoint) and the compositional risk model are patterns directly applicable to evaluation harnesses that must test non-deterministic multi-step agents. For harness builders, the paper's treatment of how to embed monitoring and mitigation logic close to risk sources—rather than at system boundaries—offers a concrete architectural pattern for runtime harness intervention.