Stealthy World Model Manipulation via Data Poisoning

🕒 Published (v1): 2026-06-17 05:24 UTC · Source: Arxiv · Venue: NEURIPS 2026 · link

Why this paper was selected

NeurIPS 2026; first systematic poisoning attack on model-based RL world models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SWAAP is a two-stage data poisoning framework that attacks learned world models in model-based RL by stealthily manipulating a small fraction of fine-tuning transitions. It first finds an adversarial target world model via bilevel optimization, then realizes it through gradient-matched poisoning that evades standard detectors. The attack causes large performance drops while keeping poisoned data indistinguishable from clean dynamics.

Problem

Learned world models are increasingly used as the planning backbone for agents in robotics and autonomous systems, yet their robustness to training-time data poisoning is largely unexplored. Standard data poisoning from supervised learning assumes discrete labels and fully differentiable pipelines; RL-based poisoning methods target rewards/policies directly rather than dynamics, and do not scale to deep latent world models with continuous high-dimensional states and long-horizon compounding errors.

Method

SWAAP decomposes the intractable bilevel poisoning problem (Eq. 1) into two stages:

Stage 1 — Target model identification. Solves a model-space bilevel problem to find a target world model \(\hat{\psi}\) that minimizes real-environment return \(J(P, \theta^*(\psi))\) while staying close to clean dynamics via regularizer \(\lambda L(P_\psi, P)\). The inner problem captures the victim planner's best response \(\theta^*(\psi)\). Because differentiating through the implicit best response is intractable, SWAAP uses a first-order dynamic-barrier method (BOME) and a novel transition-gradient theorem (Theorem 3.1): $\(\nabla_\psi J(\psi,\theta) = \mathbb{E}\!\left[\sum_{t=0}^{T-1}\gamma^t\!\left(R(s_t,a_t,s_{t+1})+\gamma V_{t+1}(s_{t+1})\right)\nabla_\psi \log P_\psi(s_{t+1}|s_t,a_t)\right]\)$ This is the policy-gradient analogue for transition parameters rather than policy parameters.

Stage 2 — Gradient-matched data realization. Selects the top-\(r_p\) fraction of transitions with the largest residual under \(\hat{\psi}\) (most inconsistent with target dynamics), then optimizes poisoned next-state targets \(\tilde{s}'\) by minimizing: $\(L_P = (1-\alpha)\!\left[1 - \cos(G_\text{real}, G_\text{target})\right] + \alpha\sum_{D_p}\!\left(\|\tilde{s}'_i - s'_i\|_2 - \|P_{\psi_0}(s_i,a_i)-s'_i\|_2\right)^2\)$ The cosine term aligns fine-tuning gradients toward Stage 1's target; the regularizer keeps perturbation magnitudes within the clean model's natural prediction-error scale. A trajectory-consistent variant additionally replaces the same poisoned latent state at both \(t\) (as \(s'_t\)) and \(t{+}1\) (as current-state input) to avoid detectable trajectory inconsistencies.

Key Contributions

• First data poisoning attack explicitly targeting learned world-model dynamics in deep model-based RL (TD-MPC2, DINO-WM, DreamerV3-class).
• Transition-gradient theorem enabling gradient-based bilevel optimization over stochastic transition models — a novel theoretical tool extending the policy-gradient theorem.
• Two-stage decoupling: model-space target search (Stage 1) + gradient-matched data realization (Stage 2); ablations confirm both stages are necessary.
• Prediction-error regularizer in Stage 2 that camouflages poisoned targets within the clean model's natural approximation noise.
• Trajectory-consistent poisoning variant that maintains latent state coherence across adjacent time steps.
• Systematic three-stage stealthiness evaluation: pre-training detection (\(\delta_d\), CUSUM), training-time robustness (TRIM), and test-time model deviation (\(\delta_m\)).

Results

• Main setting: TD-MPC2, \(r_p = 0.1\) (10% of 5,000 fine-tuning transitions poisoned), evaluated on DMControl, MyoSuite, MetaWorld, ManiSkill2.
• SWAAP causes substantial return degradation across all benchmarks; the text is truncated before the full Table 1 numbers, but the paper reports results comparable to Direct Model Poisoning (which directly overwrites the model), establishing that data-level poisoning approaches the upper bound of direct model overwrite.
• SWAAP (Random) — ablation with randomly selected poisoning targets — is less effective, confirming the value of gradient matching.
• Stealthiness: \(\delta_d\) (data-level deviation) remains close to \(\delta_{d,\text{ref}}\) (clean baseline), evading deviation-based filtering and CUSUM detectors; poisoned latent targets overlap with clean next-state clusters in PCA.
• TRIM defense: SWAAP remains effective under iterative trimming (discarding top-\(\beta\) high-residual transitions), attributed to the prediction-error regularizer keeping poisoned residuals near clean levels.
• Gray-box transfer (Appendix H.3): attack transfers without exact victim weights, broadening the practical threat.

Limitations

• White-box assumption in main experiments (victim architecture and pretrained weights known); gray-box results are preliminary appendix results, not the primary evaluation.
• Attacker requires a clean copy of the environment to collect reference trajectories for Stage 1 and stealth regularization — a non-trivial capability requirement.
• Evaluated defenses are non-adaptive (residual/CUSUM/TRIM); adaptive detectors aware of SWAAP's specific stealth strategy could be stronger.
• Stealthiness relies on operating in latent space; raw-observation transfer is addressed only with initial evidence (Appendix H.3), not a full evaluation.
• Results reported only for continuous-control MPC/model-based policy improvement; applicability to other planning paradigms (e.g., MCTS, diffusion planners) is untested.

Relevance to Agentic AI / LLM Agents

As agentic systems increasingly rely on learned world models (DreamerV3, TD-MPC2, foundation world models from OpenAI/NVIDIA) for planning and long-horizon reasoning, the security of the world-model adaptation pipeline becomes a first-class concern. SWAAP reveals that fine-tuning from locally collected experience — a standard continual-learning paradigm — is an exploitable attack surface, with adversaries able to corrupt an agent's internal model of reality without touching observed rewards or policy weights. For researchers building LLM-based agents that incorporate learned simulators or environment models (e.g., world-model-augmented tool-use agents), this work establishes the need for poisoning-aware fine-tuning protocols and motivates the development of certified robust world-model training methods analogous to certified defenses in supervised learning.