Group-in-Group Policy Optimization for LLM Agent Training

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GiGPO (Group-in-Group Policy Optimization) is a critic-free RL algorithm for multi-turn LLM agent training that introduces hierarchical credit assignment via a novel anchor state grouping mechanism. It retroactively constructs step-level advantage groups by identifying repeated environment states across rollout trajectories—requiring no extra LLM calls or value networks. On ALFWorld and WebShop it outperforms GRPO by >12% and >9% respectively with <0.002% additional compute.

Problem

Group-based RL methods (GRPO, RLOO) assign a single advantage score to an entire trajectory, which is adequate for single-turn tasks (math, code) where reward is immediate. In multi-turn agent settings with sparse or delayed rewards and episodes spanning 50+ steps and 20k+ tokens, this episode-level granularity fails to distinguish good from bad individual actions—a fundamental credit assignment problem that prior critic-based solutions (PPO, ArCHer) address only at the cost of auxiliary value networks and memory overhead.

Method

GiGPO nests two levels of advantage estimation inside a single group-based RL framework:

Episode-level (macro): Standard group normalization over N complete trajectories sharing the same task and initial state produces an episode relative advantage AE(τᵢ), identical to GRPO but with optional Fnorm=1 (Leave-One-Out) instead of std to avoid difficulty bias on hard tasks.

Step-level (micro) via anchor state grouping: After rollout, GiGPO scans all N trajectories for identical environment states. States matching across or within trajectories are termed "anchor states." For each unique anchor state s̃, it constructs a step-level group GS(s̃) = {(aₜ⁽ⁱ⁾, Rₜ⁽ⁱ⁾) | sₜ⁽ⁱ⁾ = s̃} using discounted future returns. A normalized step relative advantage AS(aₜ) is computed within this group via the same group-normalize formula. Grouping uses hashmap lookups (0.01s/iteration); no extra LLM forward passes are needed.

Combined objective: The final per-step advantage is A(aₜ) = AE(τᵢ) + ω·AS(aₜ) (ω=1 by default), fed into a clipped PPO-style objective with KL penalty. In the degenerate case where no states repeat, AS=0 and GiGPO collapses exactly to GRPO. A similarity-based variant (longest-common-subsequence threshold 0.9) handles near-duplicate states in noisy environments.

Key Contributions

• Anchor state grouping: offline, zero-rollout mechanism for constructing step-level comparison groups from existing trajectory data
• Hierarchical advantage (GiGPO): combining episode-level (AE) and step-level (AS) signals in a single critic-free objective
• Graceful degradation: reduces to GRPO when no state repeats, preserving its stability as a lower bound
• Efficiency: <0.002% additional time cost (0.54s vs 362.83s/iteration) and identical GPU memory to GRPO
• Emergent behavior: agents trained with GiGPO exhibit reduced tool redundancy (avg ~0.9 calls/query on single-hop QA) and less looping

Results

ALFWorld (success rate, Qwen2.5): - GiGPOw/o std vs GRPO: +13.3% at 1.5B (86.1% vs ~72.8%), +12.6% at 7B (90.2% vs ~77.6%) - Exceeds PPO at 7B (90.2% vs 80.4%) without a critic network - Outperforms Gemini-2.5-Pro (60.3%) by large margins

WebShop (success rate): - GiGPOw/o std vs GRPO: +10.6% at 1.5B (67.4% vs ~56.8%), +9.1% at 7B (75.2% vs ~66.1%)

Search-augmented QA (7-dataset average EM): - GiGPO 3B: 42.1% vs StepSearch 3B best baseline - GiGPO 7B: 47.2% vs StepSearch 7B; large gains on Bamboogle (68.9% vs 40.0% for StepSearch)

Ablation: Removing either AE or AS significantly degrades performance; removing AS hurts most on long-horizon tasks (Cool, Pick2, WebShop).

Step-level group dynamics: >65% of anchor states recur across trajectories throughout training; group size concentrates near N=8 at convergence, indicating policy coherence.

Limitations

• Relies on exact (or near-exact) state matching; noisy or high-dimensional observation spaces reduce the frequency of anchor state hits, weakening step-level signals
• In the extreme of fully unique states (no repeats), the step signal vanishes and GiGPO degrades to GRPO—providing no benefit over the baseline
• Similarity-based grouping (LCS threshold) is a heuristic; embedding-based or domain-specific structural equivalence remains unexplored
• Evaluated only on text-based environments (ALFWorld, WebShop, QA); applicability to visual or continuous-action agents not demonstrated

Relevance to Agentic AI / LLM Agents

GiGPO directly addresses one of the core training challenges for LLM agents: assigning per-step credit in sparse-reward, long-horizon interaction loops without the overhead of critic networks. It offers a practical and highly efficient upgrade to the GRPO/RLOO training paradigm that is already widely adopted for reasoning LLMs, making it directly applicable to web agents, embodied agents, and tool-using agents. The anchor state grouping insight—that repeated states in rollout batches are a free source of counterfactual comparisons—is a conceptually novel and reusable primitive that could be incorporated into future agent RL frameworks. For researchers tracking training-time methods for agentic LLMs, this work establishes a new efficiency-accuracy Pareto frontier on standard benchmarks.