Skip to content

Self-State Attacks on Self-Hosted AI Agents: How Far Can OS Defenses Go?

🕒 Published (v1): 2026-07-20 14:16 UTC · Source: Arxiv · link

Why this paper was selected

Schmidhuber co-author; self-state OS-level attack vector on self-hosted agents is novel and underexplored

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Self-hosted LLM agents (e.g., Claude Code, OpenClaw) must continuously read/write their own memory, persona, and configuration files to function, producing a class of threats—self-state attacks—where a compromised agent corrupts its own persistent state via operations that are formally indistinguishable from legitimate self-updates at the OS level (\(A(R) \subseteq L(R)\)). The paper formalizes this attack surface along four axes, derives structural limits for OS-layer prevention, detection, and recovery, and validates these limits empirically on a real agent under four workload profiles.

Problem

Self-hosted agents have no architectural separation between their general write capability and their self-state update paths: memory, instruction, and configuration files are ordinary files modified by the same process under the same OS principal as every other task. Existing defenses focus on the semantic layer (LLM-as-judge, guard agents) but are probabilistic and subject to adaptive adversarial paraphrase. The OS layer—the last deterministic enforcement point before a syscall commits to disk—has not been systematically studied for this attack class. It is unknown which self-state attacks the OS can close, which depend on deployment, and which necessarily escape any OS-layer mechanism.

Method

Attack space formalization. The paper defines self-state attacks via \(A(R) \subseteq L(R)\) and organizes the attack surface along four axes, each projecting onto a component of the VFS feature tuple \(\langle\text{target-path}, \text{op\_type}, \delta_s, \text{mode-delta}, \Delta t\rangle\) observable by any kernel-level monitor (inotify, fanotify, auditd):

  • D1 Target: which self-state layer is corrupted (Instruction, Memory, Config)
  • D2 Mechanism: M1 Modify (overwrite), M2 Add (append/insert), M3 Delete (unlink/truncate), M4 Deny (permission change)
  • D3 Granularity: G1 whole-file → G4 minimal-delta (≤4 B, e.g., flipping a Boolean)
  • D4 Temporal: T1 single-shot, T2 burst-aligned, T3 quiet-gap, T4 slow-drip

This yields a Target Ă— Mechanism matrix of 23 attack cells instantiated as 43 concrete operations on real self-state files (MEMORY.md, SOUL.md, AGENTS.md, openclaw.json, etc.), grounded in MITRE ATLAS (aml.t0080, aml.t0081), MITRE ATT&CK (T1565.001, T1070.004, T1222, T1562), and OWASP Agentic AI (T1, T2, T3, T6, T9).

Defense analysis. Three OS defense dimensions are analyzed—prevention (DAC, MAC, LSM, chattr), detection (inotify, auditd, FIM), and recovery (backup, CoW, CRIU)—with a formal structural limit derived for each.

Workload-conditioned detection. Because real workloads concentrate legitimate writes in a narrow region of \((\text{op\_type}, |\delta_s|, \Delta t)\) feature space, per-file baselines learned from normal traces can partially recover detection capability. This produces three cell classes: Visible (V) — anomalous under all profiles, Conditioned (C) — detectability is profile-dependent, Indistinguishable (I) — falls within the legitimate distribution of at least one profile.

Empirical setup. An OpenClaw agent (openclaw-core harness on Ubuntu 22.04) is evaluated across four workload profiles: W1 Coding assistant (Aider), W2 Research assistant (FRAMES benchmark), W3 Operations agent, W4 Generalist (30 tasks each). Attack traces are injected and evaluated against canonical and workload-conditioned detectors per cell.

Key Contributions

  • Formal definition of self-state attacks as a distinct security problem: \(A(R) \subseteq L(R)\) for writable self-state resources, making OS prevention structurally impossible without blocking legitimate behavior.
  • Four-axis attack taxonomy (Target Ă— Mechanism Ă— Granularity Ă— Temporal) that refines defender observability where existing catalogs bundle Granularity and Temporal implicitly, splitting apparently identical catalog entries into tractable vs. intractable OS-layer cases.
  • Three formal structural-limit claims: prevention indistinguishability (Claim 1), detection insufficiency (Claim 2), and recovery dependency on detection plus \(O(\Delta t)\) data loss.
  • Workload-conditioned anomaly detection as a design principle, with the V/C/I per-cell classification framework.
  • Empirical evaluation on a real self-hosted agent showing a layered defense stack (access-control on Instruction/Config layers + workload-conditioned detection on Memory layer + periodic backup) covers most of the attack matrix, with a residual structurally indistinguishable surface.

Results

  • A layered defense stack—access-control prevention on the Instruction and Config layers, workload-conditioned detection on the Memory layer, periodic backup for recovery—is effective on most attack cells.
  • A small residual attack surface is structurally indistinguishable at the OS level (I-class cells); no kernel-level mechanism closes this set.
  • A significant fraction of self-state attacks are indistinguishable from benign behavior at the syscall level (stated in contributions; per-cell breakdown in truncated Section 5).
  • Detection effectiveness depends strongly on workload context: the same write operation may be anomalous under W1 (Coding, low memory churn) but typical under W3 (Operations, high config writes), confirming no single global threshold is optimal.
  • G4 minimal-delta attacks (≤4 B, e.g., toggling a Boolean) are especially difficult because they fall within the legitimate write size distribution of most profiles.
  • T4 slow-drip attacks distribute writes below per-event thresholds; their cumulative effect evades per-event detectors.

(Note: the provided text is truncated before Section 5; specific per-cell TPR/FPR figures are not available above.)

Limitations

  • Structural prevention gap is unresolvable at the OS layer: any access-control mechanism that grants the agent write access to its own self-state cannot block self-state attacks without also blocking legitimate operation.
  • Recovery inherits detection as a precondition: without reliable detection, the poisoned snapshot cannot be identified; recovery-point ambiguity and cross-layer state entanglement (memory → instruction → config) compound this.
  • Evaluation is single-agent: experiments use OpenClaw only; generalization to other self-hosted agents (Claude Code, Aider-native, etc.) is asserted architecturally but not directly measured.
  • Prompt-injection layer is out of scope: the paper assumes compromise is already achieved and does not evaluate what fraction of injections succeed in practice.
  • Workload-conditioned baselines require offline calibration: effectiveness depends on having representative normal traces per deployment profile, which may not be available at initial deployment.
  • I-class cells (structurally indistinguishable) remain open: the framework characterizes but does not solve this residual surface.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a structural vulnerability in the dominant architecture of deployed autonomous agents: the use of mutable local files (MEMORY.md, SOUL.md, AGENTS.md, config JSON) as the agent's persistent identity and behavioral substrate. It formalizes why the confused-deputy problem is harder for LLM agents than for classical systems—natural language amplifies attacker flexibility and the poisoned state shapes unbounded future inference without OS-observable signatures. For practitioners building self-hosted agents, it provides concrete design guidance: harden Instruction and Config layers with immutable permissions, apply workload-conditioned monitoring to the Memory layer, and accept that a residual attack surface cannot be closed at the OS level, requiring semantic defenses as a complement. The V/C/I classification framework is directly actionable for evaluating agent deployments and for red-teaming memory-persistent agent systems.