LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning¶

🕒 Published (v1): 2026-07-15 06:55 UTC · Source: Arxiv · link

Why this paper was selected

Self-generated process rewards for multi-turn search; removes need for dense human annotation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LAPO addresses the credit-assignment problem in multi-turn search reasoning by introducing backward leave-one-turn attribution: for each search turn, it masks the turn with [DELETE] and measures how the policy's log-likelihood of the gold answer changes, producing a self-generated process reward without any external model. Combined with sign-consistency gating, LAPO achieves an average exact-match of 0.326 across seven QA benchmarks, outperforming the strongest step-reward baseline by 0.053.

Problem¶

Terminal outcome rewards in RL-trained multi-turn search agents assign the same credit signal to all turns in a trajectory, failing to distinguish helpful, redundant, and harmful intermediate search interactions. Existing process-reward methods (PRMs, teacher models, LLM-as-a-Judge) require additional components that are expensive, potentially miscalibrated, and dependent on a separate policy distribution.

Method¶

Backward leave-one-turn attribution. Given a complete trajectory \(\tau_i = (x_i, u_{i,1}, o_{i,1}, \ldots, u_{i,T_i}, o_{i,T_i}, v_i)\), LAPO constructs a counterfactual context \(c_{i,-t}\) by replacing turn \(t\) and its retrieval observation with a fixed [DELETE] placeholder while leaving all downstream turns intact. The Answer-Likelihood Gain is:

\[\Delta_{i,t} = S_\theta(a_i^\star \mid c_i^\text{full}) - S_\theta(a_i^\star \mid c_{i,-t})\]

where \(S_\theta(a^\star \mid c) = \frac{1}{L}\sum_{k=1}^{L} \log \pi_\theta(a^\star_k \mid c, p, a^\star_{<k})\) is the mean token log-likelihood of the gold answer.

Robust scaling and normalization. Raw gains are scaled by the within-group median absolute gain, compressed via \(\tanh\), then group-normalized to produce \(\tilde{z}_{i,t}\).

Sign-consistency gating. A gate \(g_{i,t} = \mathbb{1}[\Delta_{i,t} \cdot \tilde{z}_{i,t} > 0]\) discards turns where group normalization inverts the raw attribution sign—preventing directionally contradictory updates. The gated process advantage is \(A^{\text{proc}}_{i,t} = g_{i,t}\,\tilde{z}_{i,t}\).

Advantage assignment. Search tokens receive \(A^{\text{out}}_i + \lambda A^{\text{proc}}_{i,t}\); the final-answer turn receives only \(A^{\text{out}}_i\); environment observation tokens are excluded from optimization. Policy optimization uses GRPO's clipped surrogate objective with KL regularization.

Key Contributions¶

Identifies that outcome rewards conflate useful, redundant, and harmful search turns within a trajectory.
Proposes backward leave-one-turn attribution: evaluating a turn's contribution in the full downstream reasoning context rather than by local adjacent-state differences.
Introduces sign-consistency gating to filter normalized advantages whose direction conflicts with raw attribution.
Demonstrates that process supervision can be derived entirely from the current policy, requiring no separate PRM, teacher, verifier, or LLM judge.

Results¶

LAPO achieves average EM of 0.326 across seven QA benchmarks (NQ, TriviaQA, HotpotQA, 2Wiki, MuSiQue, Bamboogle, PopQA).
Outperforms IGPO (strongest step-reward baseline) by +0.053 average EM (0.326 vs. 0.273).
Outperforms best outcome-reward baseline (Search-R1-instruct) by +0.063 average EM (0.326 vs. 0.263).
Per-dataset gains over IGPO: NQ +0.064, TQ +0.034, HotpotQA +0.030, 2Wiki +0.076, MuSiQue +0.052, Bamboogle +0.052, PopQA +0.062.
Ablations confirm complementary benefit from both backward attribution and sign-consistency gating.

Limitations¶

Requires access to gold answers during training to compute \(S_\theta(a^\star \mid \cdot)\); inapplicable to settings without ground-truth labels at training time.
Adds inference-time cost proportional to the number of search turns per trajectory (\(T_i\) extra forward passes per rollout) for attribution scoring.
Evaluated only with local Wikipedia retrieval; generalization to live web search or tool-augmented agents is unverified.
Maximum search depth fixed at three turns; behavior under longer horizons is unstudied.
Attribution is at the turn level, not the token or sub-sentence level, which may still be too coarse for fine-grained credit within a turn.

Relevance to Agentic AI / LLM Agents¶

LAPO directly addresses the credit-assignment bottleneck that limits RL training of multi-turn tool-using agents: how to differentiate the contribution of individual agentic steps without external supervisors. The backward leave-one-turn attribution framework is domain-agnostic and could extend beyond search to any agentic loop (code execution, API calls, browser actions) where turn-deletion counterfactuals are computable. The sign-consistency gating mechanism is a practical filter for the normalization artifacts that arise when mixing process and outcome rewards—a recurring problem in GRPO/PPO-based agent training. This work belongs to the growing line of self-improving agentic RL that avoids teacher distillation, sitting between pure GRPO (no process signal) and PRM-based methods (costly external signal).