Skip to content

Building a Foundational Guardrail for General Agentic Systems via Synthetic Data

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Synthetic data pipeline for planning-stage safety guardrails; pre-action harm prevention

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper tackles pre-execution safety for LLM agent systems by introducing three components: AuraGen, a synthetic data engine that injects calibrated risks into agent trajectories; Safiron, a compact guardian model trained with SFT + GRPO that intercepts harmful plans before execution; and Pre-Exec Bench, a human-verified benchmark covering planning-stage safety across diverse tools and risk types. The unified framework consistently outperforms open-weight and proprietary baselines on Pre-Exec Bench.

Problem

Existing agentic guardrails are predominantly post-execution, leaving no proactive intervention at the planning stage. Three specific gaps stall progress: (1) data gap — real harmful agent trajectories are rare, heterogeneous, and expensive to annotate; (2) model gap — current guardian models are narrow in scope and lack cross-planner adaptivity; (3) evaluation gap — existing benchmarks (ASB, R-Judge, AgentSafetyBench) focus on execution-time risks and single-environment settings, making them ill-suited for systematic planning-stage evaluation.

Method

AuraGen (data engine) operates in three stages: 1. Benign synthesis: given metadata \(M = (E, U, C)\) (environment, tools, constraints), an LLM \(G_{\text{gen}}\) produces a user query \(q\) and benign trajectory \(T_{\text{benign}} = (a_1, \ldots, a_n)\). 2. Principled risk injection: an injection model \(G_{\text{inject}}\) transforms \(T_{\text{benign}}\) using one of four strategies — single-step perturbation (\(S_{\text{single}}\)), multi-step corruption (\(S_{\text{multi}}\)), new branch diversion (\(S_{\text{new}}\), goal hijacking), and bridged branch diversion (\(S_{\text{bridge}}\), deceptive completion). A risk category \(r\) is sampled from taxonomy \(R\). 3. Automated quality assurance: a reward model \(M_{\text{RM}}\) scores each sample on five axes (causal consistency, postcondition continuity, rationality, justification sufficiency, risk matching) and a filter policy retains or discards samples.

Safiron (guardian model) combines a cross-planner adapter (normalizes heterogeneous input formats) with a compact model trained in two stages: - Stage 1: SFT on \(\sim\)20k AuraGen samples (Ministral-8B base, data from Mixtral-8×22B). - Stage 2: GRPO-based RL on \(D_{\text{RL}} = D_{\text{easy}} \cup D_{\text{hard}}\) with reward \(R(\hat{y}, y) \in \{0, 0.5, 1.0\}\) rewarding correct binary detection and fine-grained risk categorization. The GRPO objective is: $\(\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_x \left[ \mathbb{E}_{i,t} \left[ \min\!\left(\rho_{i,t}\tilde{A}_i,\, \text{clip}(\rho_{i,t}, 1{-}\epsilon, 1{+}\epsilon)\tilde{A}_i\right) \right] - \beta\, \mathbb{E}_{i,t}\, \text{KL}_{i,t}(x) \right]\)$ Output per trajectory \(T\): \((y_{\text{risk}}, y_{\text{type}}, e) = G(T)\) — binary label, risk category, explanation.

Pre-Exec Bench is built via (1) tool refinement from AgentSafetyBench scenarios, (2) diverse trajectory generation using 8 heterogeneous LLMs from 5 developers with cross-model benign/risk decoupling, and (3) two-phase human verification (quality gate + debiasing), yielding 1,001 harmless and 671 risky samples held out from all training.

Key Contributions

  • AuraGen: controllable synthetic data engine with four injection strategies and automated reward-model filtering, producing \(\sim\)20k balanced training trajectories.
  • Safiron: compact guardian model (Ministral-8B base) with cross-planner adapter, trained via SFT→GRPO pipeline, producing interpretable three-field outputs.
  • Pre-Exec Bench: 1,672-sample human-verified planning-stage benchmark covering detection, fine-grained categorization, explanation correctness, and cross-planner generalization.
  • Training recipe analysis: shows that harmful/harmless ratio matters more than dataset size, with performance saturating around 1:4–1:6 harmful-to-benign ratio; mixing easy and hard samples in GRPO prevents collapse.

Results

  • Safiron (adapter + guardian) achieves consistent gains over both open-weight and proprietary baselines on all four Pre-Exec Bench metrics: classification accuracy, harmful detection precision, risk category accuracy, and explanation correctness.
  • AuraGen corpus achieves near-uniform coverage across four injection strategies (~25% each) with realistic query lengths (mean 23.10 tokens, median 21) and long-tailed trajectory complexity (mean 14.77 actions, median 9, max ~48).
  • Training on \(D_{\text{hard}}\) alone collapses; combining \(D_{\text{easy}} \cup D_{\text{hard}}\) in GRPO stabilizes training and yields the best performance.
  • Per-sample data generation cost with GPT-5 is under $0.02; open-source alternatives are strictly cheaper.

(Full numeric tables are not included in the provided text excerpt.)

Limitations

  • Explanation quality is not directly optimized in RL (only indirectly, via correlation with category accuracy); complex explanation evaluation would make RL training cost-prohibitive.
  • LLM-synthesized trajectories in Pre-Exec Bench cannot fully eliminate model-specific stylistic artifacts even after human debiasing.
  • Base guardian model is Ministral-8B; generalization to significantly larger or different architectures is not fully characterized.
  • Risk taxonomy is pre-defined; coverage of truly novel or emerging risk categories requires manual expansion of the injection risk pool.
  • Results on Pre-Exec Bench only; real-world deployment safety properties beyond the benchmark are not evaluated.

Relevance to Agentic AI / LLM Agents

This work directly addresses one of the most critical open problems in agentic AI: how to enforce safety proactively at the planning stage rather than reactively after harm has occurred. The pre-execution framing generalizes across agent architectures (any system with a planning phase), making the guardrail infrastructure broadly applicable rather than tool- or environment-specific. The AuraGen synthetic data engine provides a practical recipe for overcoming the data scarcity that has been a persistent blocker for training agentic safety models, and the GRPO-based fine-tuning methodology connects to the broader trend of using RL to improve LLM reasoning in agentic contexts. Pre-Exec Bench fills a concrete evaluation gap and is designed as a community resource for future planning-stage guardrail research.