Skip to content

SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do

🕒 Published (v1): 2026-03-28 05:52 UTC · Source: Arxiv · Venue: COLM · link

Why this paper was selected

COLM; safety drift — individually safe agent actions compounding into violations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SafetyDrift addresses the problem that individually safe LLM agent actions can compound into safety violations (e.g., read file → summarize → email externally = data leak). The framework models agent safety trajectories as absorbing Markov chains over a three-dimensional safety state, enabling finite-horizon violation probability computation with closed-form solutions. A lightweight category-aware runtime monitor achieves 94.7% violation detection with 3.7 steps of advance warning, running >60,000× faster than LLM-judge baselines.

Problem

Existing agent safety benchmarks (OpenAgentSafety, ODCV-Bench, Agent-SafetyBench) measure safety violations after the fact; none predict them in real time. Per-step safety monitors evaluate each action in isolation and structurally cannot detect safety drift—the phenomenon where \(\forall t, \text{safe}(a_t | \text{context}_t) = \text{true}\) yet the full trace \(\tau\) violates a safety property. As agents gain powerful tool access, the risk compounds combinatorially, with documented unsafe behavior rates of 49–73% across safety-vulnerable tasks.

Method

Safety state model. Each agent step maps to a state tuple \(s = (d, t, r)\) where: - \(d \in \{\text{NONE, PUBLIC, INTERNAL, SENSITIVE, CREDENTIALS}\}\) — maximum data exposure seen - \(t \in \{\text{READ\_ONLY, FILE\_WRITE, CODE\_EXEC, NETWORK}\}\) — highest tool capability exercised - \(r \in \{\text{FULLY\_REVERSIBLE, PARTIALLY, IRREVERSIBLE}\}\) — reversibility of cumulative actions

Both \(d\) and \(t\) are monotonically non-decreasing: \(s_{t+1} = (\max(d_t, d'), \max(t_t, t'), r')\). A deterministic synthesis function \(\rho: (d,t,r) \mapsto \ell\) maps the 60-state tuple space to five coarse risk levels \(\{\text{SAFE, MILD, ELEVATED, CRITICAL, VIOLATED}\}\) via 12 ordered rules grounded in prior safety taxonomies.

Markov chain analysis. The risk level sequence is modeled as an absorbing Markov chain. A \(5 \times 5\) transition matrix \(P\) is estimated empirically from agent traces. Because monotonicity ensures all transient states have absorption probability 1.0, the practical question is when, not if. Finite-horizon violation probability \(P(\text{VIOLATED within } h \text{ steps} \mid s_t = i) = [P^h]_{i, \text{VIOLATED}}\) is computed analytically using the fundamental matrix \(N = (I - Q)^{-1}\), yielding submicrosecond lookup at runtime.

Category-aware monitor. Separate transition matrices are fit for four task categories (data handling, sysadmin, research/comms, code debugging). At each step, a deterministic rule classifier (85% of steps) or LLM-as-judge fallback (Claude Haiku, 15%) maps the agent's action to a state delta. The monitor looks up the category-specific \(P_c(\text{VIOLATED in } h \mid s_t)\) and intervenes when it exceeds a calibrated threshold \(\theta_c\).

Dataset. 357 execution traces (2,947 steps) across 40 tasks, with Claude Sonnet in a ReAct loop. 80/20 train/test split stratified by category and violation status.

Key Contributions

  • Formal three-dimensional safety state with monotonicity constraints, giving a Markov-justified cumulative risk representation
  • Empirical absorbing Markov chain analysis revealing that (a) all agents eventually violate safety if unsupervised, (b) sharp "points of no return" exist in some categories (research/comms ELEVATED: 96.9% within 5 steps) but not others (sysadmin/code-debug: <5% from any state), and (c) these points are task-type-dependent
  • Lightweight category-aware runtime monitor using precomputed absorption probabilities — no LLM call at inference, <0.001 ms/step
  • Ablation establishing tool escalation as the most informative safety state dimension

Results

  • Detection rate: 94.7% (95% CI [83%, 99%]) vs. keyword matching 44.7%, LLM judge 52.6%, LLM judge with 3-step context 57.9%
  • False positive rate: 11.8% (95% CI [5%, 27%]) vs. keyword 55.9%, LLM judge 38.2%, LLM judge (3-step) 47.1%
  • Early warning: mean 3.7 steps in advance; distribution concentrated at 2–7 steps (mode 4)
  • Latency: <0.01 ms/step vs. >588 ms/step for LLM judge baselines (>60,000× speedup)
  • Category-specific points of no return: research/comms ELEVATED state → 96.9% violation within 5 steps; MILD → 85.1%; data handling shows gradual drift (56–67%); sysadmin and code-debug <5%
  • Markov order: 1st order 75.3% next-state accuracy vs. 3rd order 83.6%; 2nd order monitor yields identical detection/FPR as 1st order, confirming 1st order is sufficient for monitoring
  • Learning curve: accuracy plateaus ~74% around 178–357 traces

Limitations

  • Monotonicity assumption makes all absorption probabilities 1.0, so the framework answers when but not whether — this is a design choice but limits expressiveness for agents capable of safe recovery
  • Safety state operates at tool-call level; cannot inspect action content (e.g., whether an email body was sanitized), and coarse tool-level risk profiles occasionally overestimate risk for benign commands
  • Category must be known at runtime; degradation under misclassification is not quantified
  • Dataset limited to 40 tasks and 4 categories with Claude Sonnet only; generalization to other LLMs and categories (web browsing, database administration) is untested
  • 15% of step labels require LLM-as-judge fallback, introducing label noise for ambiguous actions
  • Formal Markov property is rejected by chi-squared test (p < 0.001), though practical impact on monitoring performance is modest

Relevance to Agentic AI / LLM Agents

SafetyDrift addresses a core unsolved problem in agentic deployment: trajectory-level safety, where the danger emerges from action sequences rather than individual steps—directly relevant to anyone building or auditing multi-step tool-using agents. The absorbing Markov chain formulation is a principled alternative to per-step LLM judges and provides a clear theoretical handle on when intervention is needed, complementing work on constrained MDPs and safe RL for discrete tool-call settings. The finding that task category determines whether "points of no return" exist challenges the common assumption that a universal safety monitor suffices, with direct implications for multi-agent systems where tasks span varied domains. The framework's negligible overhead makes it practically deployable as a wrapper around any existing ReAct-style agent, making it immediately applicable to production agentic systems.