Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReasonRAG applies process-supervised reinforcement learning (RL) to agentic RAG, replacing sparse outcome rewards with step-level rewards generated via Monte Carlo Tree Search (MCTS) and a novel Shortest Path Reward Estimation (SPRE) algorithm. Trained on only 5k queries, it outperforms Search-R1 (trained on 90k queries) across five QA benchmarks, demonstrating that process-level supervision is substantially more data-efficient than outcome-level supervision for multi-step retrieval agents.

Problem

Outcome-supervised RL for agentic RAG (e.g., Search-R1) suffers from three compounding problems: (1) low exploration efficiency because the model must complete an entire reasoning chain before receiving any reward signal; (2) gradient conflict when late-step errors penalize correct early-step actions; and (3) sparse rewards requiring large amounts of training data and compute to converge. No prior work had applied process-supervised RL to the RAG setting, partly because constructing step-level reward annotations for open-ended retrieval trajectories is prohibitively expensive manually.

Method

ReasonRAG builds a process-supervised training dataset (RAG-ProGuide) automatically, then trains via DPO using those process-level preferences.

SPRE (Shortest Path Reward Estimation): For a partial reasoning trajectory \(y_{1:t}\), SPRE estimates the step-level reward via Monte Carlo rollouts: $\(Q_t = \frac{1}{k}\sum_{i=1}^{k} v(\text{rollout}_i) \cdot \alpha^{\text{step}(\text{rollout}_i)}\)$ where \(v\) is F1 correctness against ground truth and \(\alpha \in (0,1]\) is a decay factor penalizing unnecessarily long paths, incentivizing correct answers reached in fewer steps.

MCTS for trajectory exploration: A tailored MCTS over three stages (Reasoning, Grounding, Terminal) and three actions (Query Generation, Evidence Extraction, Answer Generation) selects promising paths via UCB, expands via LLM sampling, and backpropagates SPRE rewards. This yields diverse, high-reward intermediate steps without human annotation.

RAG-ProGuide dataset: 4,603 questions sampled from PopQA, HotpotQA, and 2WikiMultihopQA, yielding 13,289 step-level preference pairs (chosen/rejected response at each reasoning step). Post-filtering removes duplicate pairs and pairs with reward gap < 0.01.

Policy optimization: Process-supervised DPO applied at the step level — each training example is \((x, y_{<t}, y_t^w, y_t^l)\) conditioning on prior reasoning context, rather than on the full trajectory as in standard outcome DPO.

Inference: Three-state agentic loop controlled by token markers (<query>, <evidence>, <answer>), enabling adaptive retrieval depth based on question complexity.

Key Contributions

• ReasonRAG framework: end-to-end method combining MCTS exploration with SPRE reward estimation to automatically generate process-level supervision for agentic RAG.
• SPRE algorithm: step-level reward function that jointly scores correctness and path efficiency via Monte Carlo rollouts with step-count decay.
• RAG-ProGuide dataset: 5k-query, 13k-step process-level preference dataset released publicly; designed as an off-policy dataset compatible with various LLM optimizers.
• Empirical comparison of PRL vs. ORL for RAG: systematic ablation across outcome-supervised RL (5k and 10k queries), SFT, and process-supervised RL on five benchmarks using the same backbone (Qwen2.5-7B-Instruct).

Results

• Overall (Avg. EM / F1 across 5 benchmarks): ReasonRAG 34.4% / 42.3% vs. Search-R1 32.8% / 40.7% (trained on 18× more data, 90k queries).
• HotpotQA F1: ReasonRAG 48.9% vs. Search-R1 47.0%, AutoRAG 43.7%.
• 2WikiMultihopQA F1: ReasonRAG 50.4% vs. Search-R1 48.0%.
• Bamboogle EM: ReasonRAG 36.0% vs. Search-R1 32.0%.
• MuSiQue: Only dataset where ReasonRAG (12.8% EM) does not significantly beat Search-R1 (14.6% EM).
• Training efficiency (Figure 5): ReasonRAG reaches higher EM than Search-R1 with substantially fewer GPU hours, particularly on multi-hop datasets.
• Ablation (Table 3): RL-PRL (5k) > RL-ORL (10k) > RL-ORL (5k) > SFT > Base on average; SFT hurts single-hop and out-of-domain performance.

Limitations

• MuSiQue (hardest multi-hop, out-of-domain) remains the one benchmark where Search-R1 outperforms ReasonRAG on EM, suggesting process supervision does not fully close the gap on the most compositionally complex tasks.
• MCTS-based data generation is computationally non-trivial (requires many LLM rollouts per question); the paper does not report the cost of constructing RAG-ProGuide relative to Search-R1's training cost.
• All experiments use a single backbone (Qwen2.5-7B-Instruct); generalizability to other model families or scales is unverified.
• The SPRE decay factor \(\alpha\) is a fixed hyperparameter; sensitivity analysis is not reported in the main text.
• RAG-ProGuide is constructed only from three datasets (PopQA, HotpotQA, 2WikiMultihopQA); domain coverage of the training signal is limited.

Relevance to Agentic AI / LLM Agents

This paper directly addresses the RL training problem for tool-using LLM agents, specifically in the retrieval-augmented setting, and provides a principled comparison of process vs. outcome reward shaping — a question central to training agentic systems more broadly. The MCTS+SPRE approach to automatic process-level annotation without human labelers is a practically significant contribution: it removes a major bottleneck for applying PRL to other agentic tasks beyond RAG. The result that process supervision achieves better data efficiency than outcome supervision (18× fewer training examples) has implications for any multi-step LLM agent trained with RL, including code agents, web agents, and planning systems. The three-action agentic loop (query, evidence extraction, answer) is a clean abstraction that maps directly onto general tool-use agent architectures.