Skip to content

PRInTS: Reward Modeling for Long-Horizon Information Seeking

🕒 Published (v1): 2025-11-24 17:09 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PRInTS is a generative Process Reward Model (PRM) for long-horizon information-seeking agents that jointly learns to score trajectory steps via information gain estimation and to compress growing trajectory context via recursive summarization. Applied at test-time via best-of-n selection, a 4B PRInTS model boosts diverse agents (open-source, specialized, and frontier) by 3.9–9.3% absolute accuracy across FRAMES, GAIA, and WebWalkerQA without retraining the underlying agent.

Problem

Existing PRMs—designed for short mathematical or logical reasoning—fail on long-horizon tool-augmented information-seeking tasks for two reasons: (1) they evaluate short, isolated reasoning snippets with binary correctness signals, ignoring multi-dimensional quality factors like tool call appropriateness, output interpretation, and next-action planning; and (2) they cannot handle the rapidly accumulating context (interleaved reasoning, tool calls, and lengthy tool responses) that grows across many trajectory steps, introducing noise that degrades evaluation quality.

Method

PRInTS is a unified 4B model (initialized from Qwen3-4B) with two jointly trained capabilities:

Scorer: For each candidate next step \((s_t, a_t)\), PRInTS generates a Chain-of-Thought analysis across multiple quality dimensions and outputs a dense scalar score. The score is trained to approximate the information gain—the change in Monte Carlo-estimated answer probability before vs. after the step: \(g_t = (m_t - m_{t-1}) \times M/2\). Training uses GRPO with two complementary rewards: a score reward \(r_s\) minimizing |predicted − ground-truth|, and a comparison reward \(r_c\) enforcing pairwise preferences from annotated winning/losing step pairs. An adaptive weight \(w = (g^+ - g^-)/M\) down-weights noisy pairs with small score margins, yielding the final reward \(r = r_s + w \cdot r_c\).

Summarizer: At each step, PRInTS recursively compresses the full trajectory into a compact summary \(h_t = f_S(q, h_{t-1}, o_{t-1}, s_t, a_t)\), trained via SFT on LLM-generated summary annotations. This summary replaces the raw trajectory as context for subsequent scoring, keeping input length bounded.

At test-time, \(n=4\) candidate steps are generated by the agent; PRInTS scores each using the summary as context and selects the highest-scoring candidate (best-of-n).

Key Contributions

  • A generative PRM framework (PRInTS) that simultaneously handles multi-dimensional step scoring and trajectory context compression in a single 4B model.
  • Information gain as a grounded, continuous step-quality signal estimated via Monte Carlo rollouts, replacing binary correctness for PRM training.
  • An adaptive-weight dual-reward training objective (GRPO + SFT) that mitigates noisy preference pair annotations.
  • Demonstration that model-agnostic test-time PRM guidance is a cost-efficient alternative to agent fine-tuning, generalizing across open-source, specialized, and closed frontier models without weight access.

Results

  • Qwen3-32B (Table 1): PRInTS achieves 38.8% avg. accuracy vs. 29.5% base agent (+9.3%), best across all GAIA levels, FRAMES, and WebWalkerQA; next-best baseline adds only 3.3%.
  • DeepResearch-30B-A3B (Table 2): PRInTS reaches 66.8% avg. (+3.9%); no other baseline achieves notable gains over this strong specialized agent. On GAIA: 64.4% vs. 61.9% base—outperforming DeepSeek-V3.1-671B (63.1%, 20× larger) and closing gap to OpenAI DeepResearch (67.4%).
  • Gemini-2.5-Flash (Table 3): +4.0% avg. accuracy (44.0% vs. 40.0%), +5.5% on GAIA Level 3; second-best method adds only +1.5%.
  • Context ablation (Table 4): Summary context (\(h_t\)) outperforms full raw history \(H_t\) by +7.7% and outperforms truncated last-2-steps by +3.1%.
  • Reward ablation (Table 5): Combined \(r_s + w \cdot r_c\) outperforms score-only by +3.0% and comparison-only by +4.1%.
  • Best-of-n scaling: PRInTS scales to \(n=8\) (+8.9% on GAIA L2 with DeepResearch); performance degrades at \(n=16\) due to agent preference for exploratory over concluding steps.

Limitations

  • Training data restricted to English-language QA; multilingual generalization untested.
  • Monte Carlo rollout annotation is computationally expensive (M rollouts per step), limiting data scalability.
  • Performance degrades at very large \(n\) (n=16) due to the agent abandoning near-correct trajectories for continued exploration.
  • PRInTS has not yet been used as a dense reward signal for training agents via RL (only test-time inference tested); this direction is left to future work.
  • Evaluation uses LLM-as-Judge (GPT-5), introducing potential judge bias; no human evaluation reported.

Relevance to Agentic AI / LLM Agents

PRInTS directly addresses the test-time guidance problem for tool-augmented LLM agents on long-horizon tasks—a setting increasingly central to agentic AI research (deep research agents, web agents, code agents). It establishes that PRMs can be made model-agnostic and training-free at deployment time, offering a practical plug-in component for any ReAct-style agent without modifying weights. The paper's framing of step quality as information gain estimation is a principled bridge between RL theory and practical agent evaluation, and the trajectory summarization mechanism tackles a fundamental bottleneck (context accumulation) in long-horizon agentic loops. For practitioners building or benchmarking multi-step agents, PRInTS provides a concrete recipe for test-time scaling that is more compute-efficient than scaling the backbone model.