Lyapunov Exponent as Physics-Informed Dense Reward: RL Discovery of Stabilization Beyond the Kapitza Pendulum

🕒 Published (v1): 2026-07-15 16:29 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes using the maximal Lyapunov Characteristic Exponent (LCE) as a dense reward signal for RL-based stabilization of an inverted pendulum with a vertically oscillating pivot. The LCE-trained SAC agent not only rediscovers the Kapitza pendulum but surpasses it, actively damping angular oscillations to residual amplitudes of \(\pm 0.01°\). Prior work using squared-angle reward with DDPG failed entirely to find oscillatory stabilization.

Problem

Stabilizing an inverted pendulum via vertical pivot oscillation (Kapitza pendulum) is unsolved by standard RL because the squared-angle reward \(\varphi^2\) is sparse and non-informative: if the pendulum falls mid-episode, there is no gradient signal indicating whether oscillatory motion had been on the right track. Prior DDPG attempts with \(\varphi^2\) reward either let the pendulum fall or diverged to unbounded downward acceleration.

Method

The Lyapunov Characteristic Exponent \(\nu_1\) is computed online using the Benettin renormalization algorithm over a sliding window of \(0.25\) s. The pendulum state \((y_c, \dot{y}_c, \sin\varphi, \cos\varphi, \dot\varphi)\) is recorded at each of 50 time steps within the window, forming a 250-dimensional state vector. The reward is:

\[r_t = -\nu_1 - 100 \cdot \frac{(y_c / y_{\max})^{20}}{1 + (y_c / y_{\max})^{20}}\]

The second term penalizes pivot displacement beyond \(y_{\max} = 10\) cm to prevent unbounded downward acceleration (a trivial exploit). The agent (SAC from Stable Baselines 3, two hidden layers of 256 neurons, \(\alpha = 10^{-4}\)) controls pivot vertical acceleration \(a(t)\) at 5 ms steps over 1 s episodes. An analytical derivation shows that making \(\nu_1 < 0\) requires asymmetric pivot motion with a damping term \(\eta\ddot\eta_2 \sim -\dot\delta_s\), implying the optimal policy must go beyond periodic Kapitza oscillation.

Key Contributions

• Proposes LCE as a dense, physics-informed reward that encodes long-horizon stability information unavailable in instantaneous angle-based rewards.
• Analytically derives (via averaging theory) that strict upright stabilization at \(\varphi_s = 0\) is impossible with vertical oscillation alone, since the fast correction term \(\varphi_f = -\eta\sin\varphi_s\) vanishes at \(\varphi_s = 0\).
• Shows analytically that truly negative LCE requires asymmetric, non-periodic pivot motion with a velocity-dependent damping component.
• Demonstrates empirically that LCE reward enables an SAC agent to exceed the Kapitza pendulum, achieving damped stabilization at \(\approx 0.25°\) from vertical with \(\pm 0.01°\) residual oscillations.
• Confirms the baseline failure mode: SAC with \(\varphi^2\) reward produces chaotic large-amplitude oscillations, consistent with the earlier DDPG failure.

Results

• LCE-trained agent: damps from \(10°\) initial angle (pendulum in free fall) to residual oscillations of \(\pm 0.01°\) around \(\approx 0.25°\) from vertical.
• Kapitza pendulum (same pivot frequency as trained agent): started from \(1°\) and continues oscillating without damping — qualitatively inferior.
• SAC with \(\varphi^2\) reward (coefficient \(c = 25\)): pendulum remains upright but exhibits chaotic oscillations with far larger amplitude than the LCE-trained agent.
• Training converges in \(\sim 1000\) evaluation steps; policy generalizes beyond the 2 s training episode length.

Limitations

• No quantitative benchmark against prior RL methods beyond the one DDPG/squared-angle baseline from an unpublished course project.
• The residual equilibrium angle (\(\approx 0.25°\)) is unexplained — whether it is a fundamental minimum-LCE constant or an artifact of the reward shaping is left as an open question.
• The position-penalty term in the reward is hand-tuned (\(y_{\max}\), the exponent 20, the coefficient 100), introducing design choices that may not generalize.
• Episodes are only 1–2 s, shorter than the stabilization timescale; generalization holds empirically but is not theoretically guaranteed.
• Single pendulum geometry (30 cm rod); no ablation on rod length, mass, or other physical parameters.

Relevance to Agentic AI / LLM Agents

This work is directly relevant to the reward design subproblem in agentic RL: it demonstrates that embedding physics-derived, information-theoretic signals (LCE) as dense rewards enables discovery of qualitatively superior behaviors that sparse task-completion signals cannot elicit. For LLM agents operating in complex environments, the analogous challenge is reward sparsity when success is only measurable at episode end — the LCE approach suggests a broader principle of using system-theoretic stability measures as shaping signals. The result that the agent spontaneously discovers a policy beyond the known analytical solution (Kapitza pendulum) is an example of physics-informed reward enabling open-ended capability discovery, a recurring theme in tool-using and embodied LLM agent research.