Skip to content

Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatism

🕒 Published (v1): 2025-12-04 00:07 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

NEUBAY is a model-based offline RL algorithm that abandons explicit conservatism (uncertainty penalties, short rollout caps) in favor of a Bayesian posterior over world models and history-dependent policies. It shows that long-horizon rollouts (64–512+ steps) are the key mechanism for suppressing value overestimation without conservatism, achieving new SOTA on 7 of 33 D4RL/NeoRL datasets and outperforming all prior Bayesian-inspired offline RL methods.

Problem

Most offline RL algorithms rely on explicit conservatism—penalizing out-of-dataset actions or restricting rollout horizons—to combat value overestimation. This suppresses average-case performance and prevents test-time adaptation, particularly hurting low-quality or skewed datasets where optimal actions are underrepresented. Prior Bayesian offline RL approaches fail to scale, typically reintroducing conservatism or devolving to model-free RL.

Method

NEUBAY frames offline RL as an epistemic POMDP and trains a history-dependent (recurrent) actor-critic to maximize expected return under an approximate posterior over world models (a deep ensemble of N=100 members). Five design choices enable long-horizon rollouts without conservatism: 1. Adaptive uncertainty-based rollout truncation: rollouts continue until ensemble disagreement exceeds the ζ-quantile of in-dataset uncertainty (default ζ=1.0, yielding the maximum in-dataset threshold), replacing fixed short-horizon caps. 2. Large ensemble (N=100): improves posterior fidelity under long rollouts where small ensembles diverge. 3. Layer normalization in world models: bounds predicted state norms (∄E[Ɲ_H]∄ ≀ H√k ‖W‖), giving a linear compounding error bound instead of exponential growth. 4. Real/imagined data mixing ratio Îș: balances in-dataset transitions against model-generated rollouts in the RL loss. 5. Small context encoder learning rate ηϕ: stabilizes long-history RNN representations (linear recurrent units / memoroid) to prevent exponential divergence.

Training uses recurrent off-policy RL (separate RNN encoders for actor and critic) on a mixture of real data and imagined rollouts, with a single world-model member fixed per rollout to respect the Bayesian objective.

Key Contributions

  • Demonstrates theoretically and empirically that Bayesianism outperforms conservatism on low-quality/skewed datasets via a bandit illustration with provable margin (Bayesian- vs. robust-coverage ratios).
  • Shows that long-horizon rollouts reduce value overestimation by shifting TD targets from high-bias bootstrapping toward lower-bias imagined rewards (extended Sims et al. 2024 analysis).
  • Identifies adaptive uncertainty-based truncation as the dominant driver of NEUBAY's success—collapsing to near-zero on 16 datasets at ζ=0.9.
  • Introduces layer normalization as a mechanism to bound compounding error in world models with a provable linear norm bound.
  • Provides a dataset-quality taxonomy (low-quality, medium-moderate coverage, medium-narrow coverage) that predicts when NEUBAY is preferable to conservative methods.

Results

  • D4RL locomotion (12 datasets): avg. 80.1 vs. 83.6 (VIPO, best conservative baseline); new SOTA on walker-random-v2 (27.9→34.1), walker-medium-v2 (95.5→106.4).
  • NeoRL locomotion (9 datasets): avg. 64.7 vs. 73.3 (VIPO).
  • D4RL Adroit (6 datasets): avg. 21.1 vs. 28.1 (MoMo); SOTA on pen-cloned-v1 (74.1→91.3) and hammer-cloned-v1 (5.0→14.4).
  • D4RL AntMaze (6 datasets): avg. 28.8 vs. 64.9 (LEQ); weakest regime due to sparse rewards and contact-rich dynamics.
  • Overall SOTA: 7 of 33 datasets, 3 statistically significant.
  • Rollout horizons of 64–512 steps (75th percentile) across 21/23 tasks; outperforms all Bayesian-inspired baselines (APE-V, MAPLE, CBOP, MoDAP) on most tasks.

Limitations

  • Underperforms conservative methods on medium-quality, narrow-coverage datasets (15/33 tasks), especially AntMaze where world modeling fails under sparse rewards.
  • Requires tuning two per-dataset hyperparameters (Îș and ηϕ), consistent with conservative baselines but not parameter-free.
  • Large ensemble (N=100) increases memory and compute relative to standard N=5 ensembles; ablation shows meaningful degradation at N=5 or N=20.
  • Explicit conservatism (adding λ>0) helps specific tasks (hopper-random, hopper-medium) but requires separate per-dataset tuning—no universal penalty-free or penalty-added rule.
  • AntMaze results suggest the Bayesian approach remains fundamentally limited by world-model quality under contact-rich, sparse-reward navigation.

Relevance to Agentic AI / LLM Agents

The epistemic POMDP framing—where an agent maintains a posterior over world models and adapts its policy at test time via observed history—is structurally identical to how LLM agents perform in-context learning under uncertainty. The paper's demonstration that history-dependent Bayesian agents outperform conservative ones on low-quality, skewed data is directly relevant to offline training of LLM agents from imperfect human demonstrations, where optimal actions are rare and conservatism would suppress adaptation. Long-horizon planning (64–512 steps) without explicit conservatism is a key requirement for complex agentic tasks such as language reasoning and multi-step tool use, and NEUBAY's design choices (adaptive truncation, layer normalization, recurrent memory) provide a concrete blueprint for scaling Bayesian world-model approaches to such settings. The work aligns with the emerging "test-time adaptation" paradigm in agentic AI, where agents should refine behavior as uncertainty resolves through interaction rather than being frozen by offline pessimism.