OPHR: Mastering Volatility Trading with Multi-Agent Deep Reinforcement Learning¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

OPHR is the first reinforcement learning framework for volatility trading via options, using a two-agent cooperative architecture: one agent times long/short volatility positions and another routes between pre-trained deep hedgers with varying risk aversions. Evaluated on BTC and ETH crypto options (2021–2024), it achieves superior total returns and risk-adjusted metrics over all baselines.

Problem¶

Existing RL work on options focuses on pricing and hedging in isolation, not on end-to-end volatility trading that simultaneously requires (1) timing positions based on implied volatility (IV) vs. realized volatility (RV) divergence and (2) selecting appropriate dynamic delta-hedging strategies whose optimality depends on market regime (oscillating vs. trending). Rule-based and ML forecasting approaches fail to bridge the gap between RV prediction and path-dependent PnL optimization.

Method¶

OPHR decomposes volatility trading into two cooperative MDPs:

OP-Agent (DQN with n-step TD): observes market features and outputs a ternary position decision (long/short/neutral volatility) at each hourly step. Trained with n-step TD to handle delayed, noisy rewards from theta decay and transaction costs.
HR-Agent (DQN, 1-step TD): every N steps, selects among K pre-trained deep hedgers (Murray et al. 2022) with different risk-aversion levels, based on market features, portfolio state, and Greeks (Δ, Γ, Θ, Vega). Reward is the PnL advantage over a delta-threshold baseline hedger run in a twin environment.

Training proceeds in two phases: (1) offline initialization of OP-Agent via distillation from a sub-optimal oracle policy that uses future RV/IV comparisons as labels; (2) iterative online alternating training of OP-Agent and HR-Agent so each adapts to the other's evolving policy.

Key Contributions¶

First RL framework for end-to-end volatility trading through options (not just pricing or hedging).
Two-agent cooperative architecture separating volatility timing (OP-Agent) from hedger selection (HR-Agent).
Oracle-based offline initialization to reduce exploration variance in n-step TD learning.
Iterative alternating training scheme for stable multi-agent convergence.
Empirical validation on BTC and ETH Deribit options data (2019–2024) with realistic transaction costs.

Results¶

On held-out test set (Jul 2023–Jul 2024), against 10 baselines (Long, Short, MR, MOM, GARCH, DeepVol, GBDT, MLP, LSTM, DLOT):

BTC: OPHR TR = +33.10% (next best: DLOT +4.91%, OP +21.43%); ASR = 1.87, ACR = 3.35, ASoR = 3.27 (all best); MDD = 9.41% (lowest among profitable strategies).
ETH: OPHR TR = +44.89% (next best: OP +23.49%); ASR = 1.76, ACR = 1.58, ASoR = 1.67 (all best); MDD = 27.25%.
Transaction costs: 9.36% of PnL (BTC), 5.75% (ETH) — modest due to average holding periods of 9–51 hours.
HR-Agent improves WR by 1.72–2.92% for long positions and 0.78–2.81% for short positions vs. OP-only baseline, with notable PLR gains on short positions.
All ML baselines (GBDT, MLP, LSTM, GARCH, DeepVol) achieve negative total returns on both assets.

Limitations¶

Evaluated only on cryptocurrency options (BTC/ETH on Deribit); generalization to equity or FX options markets is unvalidated.
Deep hedgers in the HR-Agent's routing set are trained on simulated data (Murray et al. 2022), not on real crypto market data, potentially limiting hedger quality.
Oracle initialization relies on future RV, which is unavailable at inference — only used offline; oracle itself is acknowledged as sub-optimal.
No exploration of volatility surface structure (smile skewness, term structure anomalies) — noted as future work.
Backtesting PnL assumes market-order execution; actual slippage in less liquid option strikes may be higher than modeled.

Relevance to Agentic AI / LLM Agents¶

OPHR exemplifies a task-decomposed multi-agent architecture where two specialized agents with distinct observation spaces, action spaces, and reward signals collaborate within a shared environment — a core design pattern in agentic AI. The alternating training scheme (each agent adapts to the other's evolving policy) is directly analogous to iterative self-improvement loops studied in multi-agent RL and LLM agent frameworks. The oracle-distillation initialization addresses cold-start exploration in long-horizon sequential decision-making, a challenge shared by agentic systems that must bootstrap from weak supervision before self-improving. For researchers tracking agentic AI, this provides a concrete case study of cooperative MDP decomposition and hierarchical action routing (HR-Agent as a meta-controller over a skill library of hedgers) in a high-stakes sequential domain.