Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Information-gain reward for RL-trained multi-turn search agents; principled formulation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IGPO (Information Gain-based Policy Optimization) addresses reward sparsity in multi-turn LLM search agents by replacing scalar outcome-only rewards with dense, turn-level rewards defined as the marginal increase in the model's ground-truth confidence at each interaction step. It combines these intrinsic information gain rewards with outcome rewards via discounted returns within a GRPO-style objective, achieving state-of-the-art results on seven QA benchmarks without requiring external reward models or Monte Carlo estimation.

Problem

Outcome-based RL (e.g., GRPO) trained on multi-turn search agents suffers from three compounding failures: (1) advantage collapse — when all rollouts in a group produce identical final answers (all correct or all wrong), group-relative advantages vanish and no gradient flows; (2) poor credit assignment — a single terminal reward cannot distinguish productive intermediate tool calls from useless ones; (3) sample inefficiency — each full trajectory yields only one scalar signal, wasting all intermediate semantic content. Prior process-reward solutions either need expensive external reward models or high-variance Monte Carlo rollout estimation.

Method

IGPO assigns an intrinsic, turn-level reward \(r_{i,t}^{IG}\) equal to the marginal ground-truth confidence gain at each interaction turn. Concretely, it teacher-forces the ground-truth answer \(a = (a_1,\ldots,a_L)\) onto the current interaction history \(C_{i,t}\) and computes the length-normalized log-probability:

\[s_{i,t} = \frac{1}{L}\sum_{j=1}^{L} \log \pi_\theta(a_j \mid C_{i,t}, a_{<j})\]

The information gain reward is then the stop-gradient difference between consecutive turns:

\[r_{i,t}^{IG} \triangleq \text{sg}(s_{i,t} - s_{i,t-1}), \quad 1 \le t < T_i\]

Information gain rewards and outcome rewards are z-normalized separately within the group to prevent scale dominance, then combined into a turn-level discounted return \(\tilde{R}_{i,t} = \sum_{k=t}^{T_i} \gamma^{k-t} \tilde{r}_{i,k}\). Each generated token receives the discounted return of its turn, replacing the rollout-level advantage in the GRPO clipped surrogate objective. Computation is made efficient via a vectorized single-forward-pass using a custom causal attention mask that appends \(T_i\) copies of the formatted ground truth, one per prefix, simultaneously.

Key Contributions

• Formal analysis of advantage collapse in outcome-reward GRPO for multi-turn settings.
• IGPO: intrinsic, ground-truth-aware, turn-level information gain rewards requiring no external reward model or Monte Carlo sampling.
• Separate group-wise normalization of information-gain vs. outcome rewards combined via discounted returns fed into a GRPO-style policy gradient objective.
• Vectorized single-forward-pass implementation for computing all per-prefix confidence scores with negligible overhead via a custom attention mask.
• Empirical demonstration of improved accuracy and sample efficiency across 7 benchmarks, especially for smaller models.

Results

• IGPO (Qwen2.5-7B-Instruct backbone) achieves 60.2 average F1 across 7 datasets (4 in-domain, 3 out-of-domain), vs.:
• GRPO: 51.9 (+8.3)
• GSPO: 52.0 (+8.2)
• PPO: 51.5 (+8.7)
• DeepResearcher (best outcome-reward RL baseline): 53.9 (+6.3)
• GiGPO (best prior step-reward method): 47.2 (+13.0)
• Per-dataset highlights: HotpotQA 59.0 vs. GRPO 48.9; 2Wiki 72.1 vs. GRPO 57.7; Bamboogle 77.0 vs. GRPO 65.1.
• Proportion of zero-advantage groups during training is substantially reduced compared to GRPO, particularly for Qwen2.5-3B-Instruct.

Limitations

• Information gain is computed using teacher-forced log-probability under the training policy itself, so the reward signal shifts as the policy updates — this introduces non-stationarity not analyzed in the paper.
• Ground-truth answer must be available at training time; the method is inapplicable to settings without ground-truth labels (e.g., open-ended generation).
• Evaluated only on English QA with web search; generalization to other tool types (code execution, databases) or languages is unvalidated.
• The discount factor \(\gamma = 1.0\) (no discounting) was used without ablation, leaving its sensitivity unexplored.
• Comparisons to StepSearch and ReasoningRAG use reported numbers from different papers under potentially different configurations.

Relevance to Agentic AI / LLM Agents

IGPO directly targets a fundamental bottleneck in training multi-turn tool-using agents: sparse terminal rewards produce degenerate gradients in long-horizon agentic trajectories. By deriving dense intrinsic rewards entirely from the model's own belief state — no external process reward model needed — it offers a practically scalable training signal for agentic search loops. The technique generalizes to any multi-turn RL setting where ground-truth answers are available, making it broadly relevant for training web-browsing, RAG, and reasoning agents. The advantage-collapse analysis also provides a useful diagnostic lens for anyone debugging agentic RL training stability.