Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Hierarchy-of-Groups PO: fine-grained stepwise RL updates for long-horizon agentic tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HGPO (Hierarchy-of-Groups Policy Optimization) addresses a fundamental bias problem in stepwise group-based RL for LLM agents: steps sharing the same current state but differing historical contexts produce unreliable advantage estimates. HGPO constructs hierarchical groups by context depth and adaptively weights their advantages, achieving state-of-the-art results on ALFWorld and WebShop without extra models or rollouts.

Problem¶

Stepwise group-based RL methods (e.g., GiGPO) cluster rollout steps by current state $s_t$ to estimate fine-grained advantages. However, steps in the same group can have different memory/history modules, causing their effective prompts to differ — a phenomenon the authors call historical context inconsistency. This biases the relative advantage estimate $A^S(\tilde{s}_i)$, degrading policy optimization. Oracle steps (same state AND same history) are too scarce to use exclusively: their groups are small (high variance) and their ratio in rollouts is low.

Method¶

HGPO adds two components on top of the stepwise framework:

Context-aware hierarchical grouping. For each step at state $s_t^{(i)}$, define the $k$-step context operator $C_k(s_t^{(i)})$ returning the $k$ preceding states. Group steps into $K+1$ hierarchical groups $G_k^H(s_t^{(i)})$ where group $k$ requires the last $k$ historical states to match: $$G_k^H(s_t^{(i)}) = \{(j,n) \in I : C_k(s_t^{(i)}) = C_k(s_n^{(j)})\}$$ This yields a nested structure $G_0^H \supseteq G_1^H \supseteq \cdots \supseteq G_K^H$, where $k=0$ degenerates to GiGPO's step-level grouping and $k=K$ approximates Oracle groups.

Adaptive weighting advantage estimation. Each hierarchical group produces an advantage estimate $A_k^H(s_t^{(i)})$ via standard group-relative normalization. The final advantage is: $$A^H(s_t^{(i)}) = \sum_{k=0}^{K} w_k A_k^H(s_t^{(i)}), \quad w_k = \frac{(k+1)^\alpha}{\sum_k (k+1)^\alpha}$$ Higher-level groups (more consistent context) receive larger weights ($\alpha \geq 0$), trading off bias against variance. Proposition 4.1 proves: $b_K \leq \text{Bias}[A^H] \leq b_0$ and $v_0/(K+1) \leq \text{Var}[A^H] \leq v_K$, interpolating between step-level and Oracle estimators. The grouping is fully offline (hashmap lookups only); no extra rollouts or models are needed.

Key Contributions¶

Identifies and empirically quantifies historical context inconsistency as a source of severe advantage estimation bias in stepwise group-based RL.
Proposes HGPO: context-aware hierarchical grouping + adaptive weighting advantage aggregation that achieves a provable bias-variance tradeoff.
State-of-the-art results on ALFWorld and WebShop under identical computational budgets (same rollouts, same GPU memory) as baselines.

Results¶

All comparisons use Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct; primary baseline is GiGPO (closest stepwise competitor).

ALFWorld (Qwen2.5-1.5B, K=2): HGPO in-dist 92.77% vs. GiGPO 90.16% (+2.61 pp); out-dist 90.16% vs. 84.76% (+5.4 pp).
ALFWorld (Qwen2.5-1.5B, K=4): HGPO in-dist 94.85% vs. GiGPO 93.29% (+1.56 pp); out-dist 92.12% vs. 91.53%.
WebShop (Qwen2.5-1.5B, K=2): Task success 71.54% vs. GiGPO 66.53% (+5.01 pp).
WebShop (Qwen2.5-1.5B, K=4): Task success 78.12% vs. GiGPO 73.24% (+4.88 pp).
Qwen2.5-7B gains are smaller (avg +0.74% at K=2, +0.13% at K=4), consistent with larger models producing shorter, less redundant rollouts with less bias.
HGPO substantially outperforms trajectory-wise RL baselines (GRPO, RLOO) and all prompting methods (GPT-4o, Gemini-2.5-Pro, ReAct, Reflexion).
Increasing $K$ improves both GiGPO and HGPO; HGPO retains superiority at all $K$.

Limitations¶

Gains are modest for larger/stronger models (Qwen2.5-7B), suggesting diminishing returns as model capability reduces rollout redundancy.
Evaluated on only two benchmarks (ALFWorld, WebShop) with two model scales; generalization to other agentic domains (GUI, software engineering, code) is not demonstrated.
The hierarchical grouping requires sufficient rollout overlap in histories; in environments with highly stochastic state transitions, $G_k^H$ for $k>0$ may degrade to Oracle sparsity anyway.
The paper is truncated before ablations and analysis of $\alpha$ sensitivity are fully described.

Relevance to Agentic AI / LLM Agents¶

HGPO directly addresses a structural flaw in the dominant paradigm for training LLM agents via group-based RL (GRPO/GiGPO), which is foundational to current agentic post-training pipelines. The insight — that stepwise RL training conflates steps with different effective prompts and thereby corrupts credit assignment — is broadly applicable to any multi-turn agent trained with group-relative methods. The hierarchical grouping idea offers a drop-in improvement to stepwise RL frameworks without changing the rollout budget, making it immediately relevant for practitioners training agents on ALFWorld-style embodied tasks or web navigation. This work also connects to the growing literature on scalable long-horizon RL (RAGEN, SearchR1) and advances understanding of when and why stepwise methods fail.