Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization¶

🕒 Published (v1): 2025-11-14 06:11 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Offline RL agents fail under data corruption because corrupted datasets induce sharp minima in the loss landscape, degrading generalization. This paper proposes applying Sharpness-Aware Minimization (SAM) as a plug-and-play optimizer replacement to find flatter minima, making agents more robust without modifying algorithm logic. SAM-enhanced IQL and RIQL consistently outperform their baselines on D4RL benchmarks under random and adversarial corruption.

Problem¶

Offline RL algorithms—including robustness-targeted ones like RIQL—still fail under observation corruption and mixture corruption (combined observation+action+reward noise). The hypothesis is that corrupted data creates sharp, unstable minima in the value function loss landscape, causing sensitivity to input perturbations and poor policy generalization.

Method¶

SAM replaces the standard Adam optimizer for the value function network only (determined by ablation). SAM's two-step minimax procedure: (1) ascent step computes an adversarial weight perturbation ε̂(θ) = ρ · ∇L(θ)/‖∇L(θ)‖₂ and moves to perturbed weights θ' = θ + ε̂(θ); (2) descent step computes the gradient at θ' and updates the original parameters. This penalizes sharpness by minimizing the worst-case loss within a ρ-radius neighborhood. SAM is applied only to Vf; Q-function and actor retain standard Adam. Ablations show V-function-only SAM yields the best gains (+3.0 avg over RIQL baseline); applying SAM to actor or Q-function alone hurts performance.

Key Contributions¶

First application of SAM as a general-purpose, plug-and-play optimizer for offline RL under data corruption.
Theoretical framing: data corruption → sharp loss landscape → poor generalization; SAM → flat minima → robustness.
Empirical demonstration that SAM selectively applied to the value function is the effective configuration.
Reward surface visualizations confirming SAM produces geometrically smoother solutions.
Evaluation across random and adversarial corruption, multiple environments, and varying corruption rates/ranges.

Results¶

Random corruption (D4RL medium-replay, 30% corruption rate): - IQL avg: 34.47 → IQL+SAM avg: 44.40 (+28.8%) - RIQL avg: 33.97 → RIQL+SAM avg: 39.47 (+16.2%)

Adversarial corruption: - IQL avg: 22.45 → IQL+SAM avg: 36.03 (+60.5%) - RIQL avg: 38.20 → RIQL+SAM avg: 40.09 (+5.0%)

Largest gains on Hopper with random mixture corruption: IQL 16.33 → IQL+SAM 55.34 (~3.4×).

Varying corruption rates (HalfCheetah, observation, ε=1.0): IQL+SAM avg 26.56 vs IQL avg 20.66 across rates 0.2–0.5.

Limitations¶

Experiments limited to three D4RL MuJoCo environments (halfcheetah, walker2d, hopper) with medium-replay datasets; generalization to other dataset types or environments not shown.
SAM requires two forward-backward passes per update step, roughly doubling value function training compute.
ρ (neighborhood radius) is environment-dependent and requires tuning; no principled selection method is provided.
Only 30% corruption rate tested in main results; ablations on varying rates are single-environment only.
Student abstract format—no comparison against uncertainty-based (UWMSG, TRACER) or data-recovery (ADG) baselines.

Relevance to Agentic AI / LLM Agents¶

Robust offline learning from corrupted demonstrations is directly relevant to agents trained on real-world interaction logs, which are frequently noisy, mislabeled, or adversarially perturbed (e.g., reward hacking, sensor noise, human feedback noise). The plug-and-play SAM optimizer approach offers a lightweight robustness improvement that requires no architectural changes, making it applicable to any offline RL backbone used in agent training pipelines. The loss-landscape geometry framing—corruption induces sharpness, which propagates to policy brittleness—connects to broader concerns about stability and distribution shift in agents trained offline and deployed in open-ended environments. This complements alignment and RLHF research where preference data quality is imperfect.