HiPRAG: Hierarchical Process Rewards for Efficient Agentic Retrieval Augmented Generation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Hierarchical process rewards improve suboptimal search in agentic RAG

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HiPRAG introduces a hierarchical process reward into RL training for agentic RAG to address over-search (redundant retrievals) and under-search (missed necessary retrievals). By parsing reasoning trajectories into discrete steps and applying an on-the-fly LLM-based judge to flag suboptimal search decisions, it achieves state-of-the-art QA accuracy while reducing over-search rate from >27% to 2.3%.

Problem

Agentic RAG agents trained with outcome-only RL rewards exhibit two systematic inefficiencies: over-search (issuing redundant queries for knowledge the model already holds) and under-search (skipping retrieval when external knowledge is needed, causing hallucinations). Existing mitigations—length penalties, confidence thresholds, or separately trained process reward models—are coarse proxies that cannot provide step-specific feedback on individual retrieval decisions.

Method

HiPRAG consists of three interlocking components:

Structured trajectory format. The agent is trained to emit reasoning inside a single <think> block containing a sequence of <step> blocks. Search steps are tuples \(s^R_i = (r_i, q_i, c_i, o_i)\) (reasoning, query, context, conclusion); non-search steps are \(s^{NR}_i = (r_i, o_i)\). This XML schema allows rule-based step segmentation without LLM post-hoc parsing during training.

On-the-fly suboptimal-search detection. For each search step, the policy model is prompted with the search query \(q_i\) in isolation to regenerate an answer \(o'_i\); an external LLM judge (GPT-4.1 mini) checks semantic equivalence of \(o_i\) and \(o'_i\)—equivalence flags over-search. For each non-search step, a verifier (GPT-5 mini) checks the factual/logical correctness of \((r_i, o_i)\); incorrectness flags under-search (the model hallucinated instead of retrieving).

Hierarchical reward function. Let \(A(T)\!\in\!\{0,1\}\) be answer correctness, \(F(T)\!\in\!\{0,1\}\) be format correctness, \(N(T)\) total steps, and \(N_\text{corr}(T)\) the count of steps flagged as neither over- nor under-search. The merged reward is: $\(R(T) = A(T)\!\left(1-\lambda_f\right) + \lambda_f F(T) + \lambda_p\, A(T)\,F(T)\,\frac{N_\text{corr}(T)}{N(T)}\)$ with \(\lambda_f=0.2\), \(\lambda_p=0.4\). The process bonus \(\lambda_p \frac{N_\text{corr}}{N}\) is gated on both answer and format correctness, so the agent is never penalized on process quality unless it first produces a correct, well-formed answer. The approach is compatible with both PPO and GRPO.

Key Contributions

• Hierarchical, knowledge-grounded process reward that decomposes into format, outcome, and step-optimality components in a single scalar.
• Efficient on-the-fly over/under-search detection via direct LLM prompting (not re-generation or separately trained reward models), runnable in parallel across a batch.
• Parsable structured output format enabling rule-based step isolation without expensive post-hoc LLM interpretation during RL rollouts.
• Demonstrated generalizability across two model families (Qwen2.5, Llama-3.2), two sizes (3B, 7B), two RL algorithms (PPO, GRPO), and base vs. instruct variants.

Results

• Accuracy (Avg. CEM, 7 benchmarks): HiPRAG-3B: 65.4%; HiPRAG-7B: 67.2% vs. best baselines β-GRPO (62.5%) and R1-Searcher++ (62.1%).
• Over-search rate: Reduced from >27% (β-GRPO, Search-R1) to 2.3% with HiPRAG.
• Under-search rate: Reduced to as low as 29.0% (from higher baseline levels).
• Average retrieval calls per question: 1.75 (HiPRAG) vs. 2.45 (Search-R1) and 2.15 (β-GRPO)—a 29% and 19% reduction respectively.
• Per-dataset best scores: NQ 71.2, TriviaQA 76.3, PopQA 66.3 (3B), HotpotQA 62.4, 2Wiki 71.7, MuSiQue 34.1, Bamboogle 52.8 (all HiPRAG-7B).
• Format-only ablation (Search-R1-step* and β-GRPO-step*) shows the structured format alone yields modest gains; the process reward provides additional improvement.

Limitations

• Over/under-search detection during training requires external proprietary LLM calls (GPT-4.1 mini, GPT-5 mini), adding latency and cost to the RL rollout loop.
• Correct-but-incomplete non-search steps are intentionally not penalized (to avoid over-search bias), so certain under-search cases are silently ignored.
• Detection accuracy depends on the quality of the external judge; imperfect judges can introduce noisy reward signals.
• Evaluation is restricted to English QA benchmarks; generalization to other task types (code, tool-use beyond search) is not demonstrated.
• Training requires 4×A100 80 GB GPUs; the added detection step increases per-step training compute relative to outcome-only baselines.

Relevance to Agentic AI / LLM Agents

HiPRAG directly addresses a core challenge for tool-using agents: learning when to invoke a tool, not just how to use it. The hierarchical process reward framework is a principled alternative to outcome-only RL (e.g., GRPO/PPO with sparse terminal rewards) for any agent that must balance internal parametric knowledge against external tool calls. The structured trajectory decomposition and step-level reward gating are techniques transferable to other tool-use settings (e.g., code execution, API calls), making this work a methodological contribution to the broader efficient-agent training literature. It also highlights the practical importance of detecting both directions of tool-use suboptimality—over- and under-invocation—which prior work on agentic efficiency (length penalties, confidence thresholds) addressed only asymmetrically.