Skip to content

Understanding and Improving Hyperbolic Deep Reinforcement Learning

🕒 Published (v1): 2025-12-16 08:49 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Hyperbolic geometry can embed the hierarchical state-space structure of RL environments with lower distortion than Euclidean space, but prior hyperbolic RL agents suffer severe optimization instabilities. This paper formally identifies large embedding norms in the Poincaré Ball and Hyperboloid models as the root cause of PPO trust-region violations, then introduces HYPER++, which fixes these via RMSNorm + learned feature scaling, the Hyperboloid model, and a categorical value loss, achieving +52% test return on ProcGen and ~30% wall-clock speedup.

Problem

Existing hyperbolic deep RL agents (e.g., Cetin et al. 2023's Hyper+S-RYM) are unstable: embedding norms grow during training, causing conformal-factor explosion in the Poincaré Ball and exponential Jacobian blow-up in the Hyperboloid, both of which produce trust-region violations in PPO despite its clipping mechanism. Prior stabilization via SpectralNorm is shown to be insufficient—it must be applied to every encoder layer, globally limiting capacity, and still fails empirically. No formal gradient-level analysis existed for hyperbolic PPO.

Method

HYPER++ is a hybrid Euclidean-hyperbolic PPO/DDQN agent with three targeted components addressing three failure modes:

  1. RMSNorm + learned feature scaling: Applied before the encoder's final activation. RMSNorm (not LayerNorm, to avoid mean-centering that distorts hyperbolic structure) provably bounds Euclidean embedding norms and the conformal factor (Proposition 4.2). A learnable sigmoid-gated scalar ξ_θ then rescales embeddings to recover usable hyperbolic volume (up to radius α=0.95 vs. 0.76 from RMSNorm alone), providing a factor of ~10³ volume gain at d=32.

  2. Hyperboloid model instead of Poincaré Ball: The Hyperboloid MLR has no conformal factor, eliminating the dominant gradient instability source. Corollary 4.3 shows RMSNorm+scaling bounds the Hyperboloid's time component x₀, preventing the exponential Jacobian growth from Eq. 6.

  3. Categorical value loss (HL-Gauss): Replaces MSE regression for the critic. Hyperbolic MLR outputs hyperplane distances, making classification-oriented categorical targets a geometrically consistent fit; MSE is a mismatch. This stabilizes the loss gradient (first term in the chain-rule decomposition) under nonstationary RL targets.

Gradient analysis is conducted in closed form for both Poincaré Ball (HNN++ MLR) and Hyperboloid exponential maps, showing that the (1 − c‖x_H‖²)² denominator in the Poincaré gradient and the sinh/cosh growth in the Hyperboloid Jacobian both diverge with growing norm ‖x_E‖.

Key Contributions

  • Formal gradient analysis of PPO instability in both Poincaré Ball and Hyperboloid models, linking large-norm embeddings to trust-region violations
  • Proof (Proposition 4.2) that RMSNorm before the last encoder activation is sufficient to bound norms and conformal factors, with a dimension-independent guarantee
  • Learned feature scaling layer (Eq. 10) recovering hyperbolic volume lost to RMSNorm without sacrificing stability
  • HYPER++: a complete stabilized hyperbolic agent generalizing across PPO, PPG, and DDQN
  • Empirical ablations confirming each component's necessity and their synergistic interaction

Results

  • ProcGen (PPO, 25M steps, 16 environments): HYPER++ achieves +52.3% relative improvement in mean test reward over Hyper+S-RYM; wins 11/16 games on test set head-to-head
  • ProcGen (PPG): HYPER++ with PPO surpasses Hyper+S-RYM with PPG; Hyper+S-RYM with PPG is substantially worse than the Euclidean baseline
  • Atari-5 (DDQN, 10M steps, 5 seeds): HYPER++ strongly outperforms both Euclidean and Hyper+S-RYM baselines across all 5 games in median, IQM, mean, and optimality gap
  • Wall-clock time: ~30% reduction in forward-pass time vs. prior hyperbolic agents (SpectralNorm eliminated)
  • Ablations: Removing RMSNorm causes complete learning failure; SpectralNorm (full or penultimate-only) also fails to learn; replacing Hyperboloid with Poincaré Ball causes modest IQM drop (0.40 → 0.34); replacing HL-Gauss with MSE degrades performance; Euclidean+full-HYPER++ method reaches IQM=0.35 vs. HYPER++ (Hyperboloid) IQM=0.40

Limitations

  • Analysis is optimization-centric; does not characterize what hierarchical structures hyperbolic representations actually capture in RL
  • No guidance on which environments benefit most from hyperbolic geometry
  • Interaction between hyperbolic geometry and other RL algorithm families (beyond PPO, PPG, DDQN) is unexplored
  • Experiments limited to ProcGen and Atari-5; generalization to continuous-control or language-grounded RL untested

Relevance to Agentic AI / LLM Agents

Agentic AI systems that plan over long horizons—tool-using agents, tree-search planners, multi-step reasoning agents—generate inherently hierarchical state spaces where hyperbolic geometry is a natural representational fit. This paper removes a key practical barrier (optimization instability) that previously made hyperbolic representations impractical in deep RL, a core engine for training agents. The categorical value loss findings and normalization insights also apply to RL-from-human-feedback and RLHF-style training pipelines where hyperbolic embeddings could improve token-sequence hierarchy modeling. More broadly, the formal gradient analysis methodology is a template for diagnosing optimization failures in non-Euclidean representation layers that may appear in future agent architectures.