Skip to content

The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works

🕒 Published (v1): 2026-07-23 12:50 UTC · Source: Arxiv · link

Why this paper was selected

Dense reward collapse under GRPO for LLM agents; identifies what actually works—critical RL training insight

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Dense per-step observation-prediction rewards, when injected into the GRPO reward channel, reliably drive LLM agents into a degenerate "dark room" absorbing state (prediction accuracy →1.0, task success →0) across all tested model scales. The root cause is GRPO's std-normalization of group advantages, which makes bounded shaping rewards produce full-scale gradient pressure in all-fail groups regardless of the shaping coefficient \(\lambda\). Switching the same signal to an auxiliary cross-entropy loss channel recovers ~20 points of task success, and a shuffled-gold placebo demonstrates that exact content–context pairing is not the active ingredient.

Problem

Sparse-reward, long-horizon LLM agents (e.g., on ALFWorld) learn slowly. A natural remedy is dense per-step supervision via observation-prediction rewards. The paper shows this remedy catastrophically fails under GRPO: the optimizer builds a "dark room" — a predictable absorbing state — and the failure is irreversible. The puzzle is that the shaping reward is return-level safe (episode distortion \(\leq \lambda\)), yet every run collapses.

Method

  • Setting: Qwen3-1.7B/4B/8B trained with GRPO on ALFWorld using a step-independent multi-turn rollout over 30–50 step horizons; group size 4, sparse success reward (10/0).
  • Prediction-sufficiency (PS) signal: At each step the agent emits a <predict> block over a task-agnostic feature schema \(\Phi\) (location, object visibility, receptacle state), rule-verified against the next environment observation. Per-step reward is the potential difference \(r_\text{pred}(t) = \lambda(\Phi_t - \Phi_{t-1})\), \(\lambda=0.1\), injected on the last response token; episode sums telescope so \(|\sum_t r_\text{pred}(t)| \leq \lambda\).
  • Theoretical analysis: In an all-fail group (all task returns equal to constant \(C\), accumulated shaping \(s_i\)): $\(\hat{A}_i = \frac{\lambda(s_i - \bar{s})}{\lambda\sigma_s + \epsilon} \xrightarrow{\lambda\sigma_s \gg \epsilon} \frac{s_i - \bar{s}}{\sigma_s}\)$ which is independent of \(\lambda\) (Proposition 1). Proposition 2 (variance-profile criterion): the danger of a dense signal under std normalization is governed by its within-group variance at saturation, not its magnitude or density.
  • Single-factor rescue: Toggling one flag — mean-only vs. mean+std normalization — is the sole difference between 0% and 51.6% success.
  • Signal-delivery matrix: Nine arms on Qwen3-4B/ALFWorld, holding signal content fixed while varying consumption mechanism (reward channel variants vs. auxiliary CE loss) and normalization.
  • HiddenRule-Gym (HRG): Synthetic POMDP with exactly computable feature coverage \(C = I(\Phi; s)/H(s)\), used to separate coverage, dynamics, and capacity failure axes.

Key Contributions

  • Failure phenomenology: "Honeymoon → turn → absorbing state" collapse replicated at 1.7B/4B/8B; terminal triple (pred accuracy 1.0, episode length pinned at horizon, success 0); post-collapse entropy rebound matching reference-model entropy (KL-pullback signature); irreversibility under annealing and prolonged training.
  • Causal localization + theory: Single-factor rescue (removing only norm_adv_by_std flips 0%→51.6%); Proposition 1 proves \(\lambda\)-invariance in all-fail groups; explains why cosine annealing fails.
  • Variance-profile criterion (Proposition 2): Signal-side safety taxonomy: signals whose within-group variance vanishes at mastery are "amplifier-safe"; potential-difference scoring is not, \(\Delta\text{acc}\) progress signals are.
  • Channel attribution: Controlled 9-arm matrix with same signal content; auxiliary-loss arms outperform all reward-channel variants by ~20 points; shuffled-gold placebo matches true-gold auxiliary loss, showing exact content–context pairing is not the active ingredient.
  • Three-axis failure map: Coverage (\(C = I(\Phi;s)/H(s)\)) × dynamics (normalization regime) × capacity, separated on HRG with a greedy mask coverage ladder.
  • Early-warning diagnostics: Joint precursor signature (entropy decline \(\wedge\) prediction saturation \(\wedge\) length pinning) hardened by three false-alarm counterexamples; univariate entropy alarms are insufficient.

Results

  • Collapse: Std-normalized PS reward collapses all three model scales to 0% success; baselines are 1.7B: 20.9%, 4B: 49.5%, 8B: 32.8%.
  • Single-factor rescue (4B): Mean-only normalization → 51.6% (peak 75.0%); no-signal mean-only control → 52.6%; net attribution of PS reward in the rescued channel ≈ 0 (−1.0pt difference, within single-seed noise).
  • Decoupling insufficient: GDPO-style per-channel decoupling with capped contribution → 31.2% (~18pt below mean-only arms).
  • Auxiliary-loss channel (4B): Teacher-forced CE on gold predictions → 69.3%; shuffled-gold placebo → 76.0% (both beating every reward-channel variant by ~20 points).
  • Advantage-scale dose–response: All-fail-group advantage magnitude: \(\pm0.01\) (harmless) / \(\pm0.07\) (chronic drag, decoupled) / \(\pm5\)–\(7\) full-scale (lethal).
  • Saturation race: Collapse ordering by scale is 1.7B@step 30 < 8B@48 < 4B@85 (non-monotonic; refutes naive "weaker model collapses earlier" prediction).
  • HRG coverage sweep (1.7B): Doubling coverage (\(C\): 0.233→0.483) more than triples above-floor success (+4.8→+16.3pt); covering supervision at 1.7B (24.0%) nearly matches pure GRPO at 4B (26.6%).

Limitations

  • All endpoints are single-seed; formal multi-seed replication and group-size controls are preregistered but not yet complete.
  • Evaluation uses last-6 validation means with ±8.4pt single-point noise (±3.4pt SE); several comparisons are described as "descriptive" due to within-noise-floor differences.
  • Coverage sweep (HRG) has only 2 of 5 planned rungs completed.
  • The variance-profile criterion is validated via retrodiction + two preregistered prospective tests within the same paper; independence from the data used to derive it is limited.
  • All claims are explicitly scoped to GRPO-family (group-relative) advantage estimation; the paper deliberately does not test critic-based PPO, where the all-fail-group degeneracy has no direct analogue.
  • Compatibility check against published reward-channel successes is not an independent test.

Relevance to Agentic AI / LLM Agents

This paper provides a mechanistic account of why a widely-proposed technique — dense per-step world-model rewards — destroys GRPO-trained agents, and identifies the precise failure point (std normalization in all-fail groups) that is invisible at the return level. The variance-profile criterion gives practitioners a principled, preregistered signal-selection rule for any dense auxiliary reward under GRPO, directly relevant to the growing class of multi-turn agentic RL systems (ALFWorld, web agents, tool-use agents). The channel attribution finding — that the same world-model signal is harmful as a reward but beneficial as an auxiliary loss — sharply reframes the design space for long-horizon agent training, and the irreversibility result means fixes must be preventive, raising the stakes for reward design choices in deployed agentic pipelines.