Skip to content

State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

🕒 Published (v1): 2026-06-25 13:39 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

State representation is a first-class design choice in RL-based energy storage trading, not a preprocessing detail. Using a fixed Double DQN agent on a pumped-hydro arbitrage environment, the paper shows that single feature families (absolute prices, relative prices, or forecasts) generalize poorly, while combining all three reaches 55.6% of optimal profit on a same-market test set and a 47.5% cross-zone median across 39 ENTSO-E bidding zones.

Problem

RL agents for energy storage arbitrage commonly overfit to the price regime seen during training: absolute price/calendar features validate well but fail on later out-of-distribution markets (negative prices, volatility spikes, different zones). No controlled study had isolated the effect of feature family choice while holding the environment, algorithm, and training protocol fixed.

Method

The testbed is HydroDam, a pumped-storage environment with 3 discrete actions (pump/hold/release), reservoir dynamics (\(mgh\) energy conversion with efficiencies 0.8/0.9), and reward equal to immediate trading revenue \(r_t \in \{-p_t \Delta E_t / 0.8,\ 0,\ 0.9\,p_t \Delta E_t\}\). Performance is normalized to dataset-specific optimal profit computed via dynamic programming over 17 reachable reservoir states.

A single Double DQN (1024-1024-512 FC, LeakyReLU, Huber loss, \(\gamma=0.99\), replay buffer, batch 2048) is trained on Belgian day-ahead prices 2007–2009, selected on 2010–2011 validation, then evaluated on: 1. Same-market test set: Belgian prices 2012–2025 (includes energy crisis, negative prices). 2. Cross-zone transfer: 39 held-out ENTSO-E bidding zones (zero-shot).

Three feature families are ablated in all \(2^3\) combinations: - Absolute: price normalized by training 99.9th percentile, log-price, weekend flag, hour/day-of-week sine-cosine encodings. - Relative: 24h/168h price z-scores, rolling price-score quantiles, relative calendar features (per-hour/day/month recency ranks), RSI, Bollinger-band position, price-to-MA ratios. - Forecast: 24-step-ahead recursive ensemble of 5 LightGBM models (MAE 5.58 EUR/MWh on validation, 15.22 on test), future price z-score, ensemble uncertainty estimate.

The non-RL baseline is a rolling price-score heuristic: pump when \(s_t = (p_t - q_{16,t})/(q_{68,t} - q_{16,t}) \leq 0\), release when \(s_t \geq 1\), using a 34-hour rolling window.

Key Contributions

  • Controlled ablation of all 7 feature-family subsets on a fixed DDQN agent and environment, isolating state representation as the sole variable.
  • Quantitative evidence that validation-set performance is a misleading proxy for generalization: absolute-only achieves 78.9% on validation but 28.8% on the test set.
  • Zero-shot cross-zone evaluation on 39 ENTSO-E markets, demonstrating that only combined feature sets reliably beat a simple rolling-quantile heuristic.
  • Release of the HydroDam environment and codebase as a reproducible benchmark.

Results

  • Absolute-only: validation 78.9%, test 28.8%, cross-zone median 5.7% — strong overfitting.
  • Relative-only: test 10.3%, cross-zone median 1.3% — hides absolute trading opportunity size.
  • Forecast-only: test 33.4%, cross-zone median 26.8% — best single family but still below heuristic (29.2% median).
  • Absolute + relative: test 49.9%, cross-zone median 39.8% — first combination to reliably beat heuristic.
  • Absolute + forecast: test 49.5%, cross-zone median 40.0%.
  • Absolute + relative + forecast: test 55.6%, cross-zone median 47.5%, beats heuristic in 38.2/39 zones on average; best single-policy in 34/39 zones.
  • Nordic zones (NO1–NO5, SE1, SE2) remain hardest: small within-day spreads reduce arbitrage opportunity for all policies.

Limitations

  • Environment is simplified: no natural water inflow, no market impact, fixed turbine efficiencies, no ancillary-service revenues.
  • Only 3 discrete actions; continuous or finer-grained action spaces are not tested.
  • Forecaster is trained on the same training set as the RL agent; relative price information leaks into the forecast block, confounding the independent contribution of forecast features.
  • Policies are trained exclusively on Belgian data; generalization is evaluated zero-shot but training distribution is single-market.
  • Nordic zones and 14 zones below 40% optimal profit show the approach is not universally solved.
  • No hyperparameter search or architecture search across feature sets — the fixed DDQN may not be optimal for all representations.

Relevance to Agentic AI / LLM Agents

Although this paper uses a narrow DDQN agent rather than an LLM, it directly addresses the observation/state design problem central to all agentic systems: what information to surface to the agent at each decision step strongly determines whether the policy generalizes across environments. The finding that validation reward is a deceptive signal for out-of-distribution robustness mirrors known problems in LLM agent benchmarking, where in-distribution task performance routinely overstates real-world capability. The controlled ablation methodology — fixing everything except the agent's input representation — offers a clean experimental template applicable to agentic LLM evaluation. The distinction between absolute context, relative/recency context, and forward-looking (forecast) context maps onto retrieval and context-window design choices in tool-using agents.