Skip to content

RSPO: Reward-Swap Policy Optimization for Multi-Turn LLM Agents

🕒 Published (v1): 2026-07-06 06:32 UTC · Source: Arxiv · link

Why this paper was selected

Reward-swap policy optimization for sparse multi-turn agent RL; directly applicable training method

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RSPO (Reward-Swap Policy Optimization) addresses the tension between sparse outcome rewards and dense process rewards in multi-turn RL for LLM agents. It uses a cyclic two-agent framework where a dense-reward-trained Agent B generates diverse trajectories that are stored with their outcome rewards and replayed to train Agent A, combining exploration breadth with optimization fidelity. On WebShop and ALFWorld, RSPO yields consistent gains of up to 12% over PPO, GRPO, and GiGPO baselines.

Problem

Multi-turn LLM agent tasks with only terminal outcome rewards suffer from sparse signals and slow convergence; algorithms like GRPO cannot learn from successful trajectories not sampled during training. Dense process rewards accelerate convergence but risk reward misalignment ("reward hacking"), biasing policy optimization away from the true objective.

Method

RSPO is a cyclic off-policy framework operating on two agent snapshots:

  1. Agent B creation: Start from Agent A; fine-tune with dense process rewards for \(k\) steps to obtain Agent B, whose behavioral space is less constrained than outcome-reward-only training.
  2. Trajectory collection: Agent B explores and collects trajectories \(\tau_i\); rewards are reverted to ground-truth outcome rewards \(R_i\) and stored in a replay buffer \(\mathcal{D}\).
  3. Agent A update: Agent A is updated via a mixed objective: $\(J_{\text{Total}}(\pi_\theta) = (1-\alpha)J_{\text{On}}(\pi_\theta) + \alpha J_{\text{Off}}(\pi_\theta)\)$ with \(\alpha = 1/8\) (off-policy fraction). Off-policy data uses a clipping center shift (Generalized Clipping Mechanism): $\(\text{clip}\!\left(\frac{\pi_\theta(a|s)}{\pi_B(a|s)},\, \frac{\pi_{\theta_{\text{old}}}(a|s)}{\pi_B(a|s)} \pm \epsilon\right)\)$ to prevent erroneous clipping of data from the distribution-shifted Agent B.

The dense process reward model is an MLP head on a frozen LLM (\(\text{Llama-3.2-3B}\)), predicting step-level rewards \(\hat{r}_t = \tanh(\text{MLP}(h_t))\) and trained with MSE against trajectory-level outcome rewards. Replay uses reward-based sampling (highest-reward trajectories prioritized). Agent A is always the final deployed model, ensuring outcome-reward consistency.

Key Contributions

  • Reward-swap mechanism: decouples exploration (dense-reward-guided Agent B) from optimization (outcome-reward-based Agent A), resolving the misalignment problem without discarding dense rewards entirely.
  • Generalized clipping for off-policy RL: adapts PPO/GRPO's proximal clipping to handle the distributional shift between Agent A and Agent B data.
  • Reward-based replay sampling: empirically superior to random or variance-based sampling; prioritizes successful trajectories to avoid instability from negative-heavy replay early in training.
  • Algorithm-agnostic wrapper: RSPO is demonstrated as a drop-in enhancement for GRPO, PPO, and GiGPO.
  • Inconsistency analysis: shows that training solely with dense rewards causes reward hacking (success rate declines while dense reward increases), motivating the swap design.

Results

  • ALFWorld, Qwen2.5-1.5B: RSPO+GRPO 74.7% vs. GRPO 69.0% (+5.7%); RSPO+PPO 76.6% vs. PPO 68.0% (+8.6%); RSPO+GiGPO 90.4% vs. GiGPO 88.8% (+1.6%).
  • WebShop, Qwen2.5-1.5B: RSPO+GRPO 70.6% SR vs. GRPO 65.1% (+5.5%); RSPO+PPO 69.3% vs. PPO 57.3% (+12.0%); RSPO+GiGPO 75.3% vs. GiGPO 67.4% (+7.9%).
  • ALFWorld, Qwen2.5-7B: +1.8% (GRPO), +2.9% (GiGPO), +2.4% (PPO); improvements smaller due to stronger baseline diversity.
  • WebShop, Qwen2.5-7B: +6.5% (GRPO), +2.0% (GiGPO), +3.4% (PPO).
  • Ablation (WebShop, 1.5B): Reward-based sampling 70.6% SR vs. variance-based 63.0% vs. random 12.8%.
  • Ablation (data source): RSPO with self-historical trajectories (RSPO\(_H\)) 57.0% SR, outcome-reward-only replay (RSPO\(_O\)) 64.1%, full RSPO 70.6%.
  • SPEAR+GRPO/GiGPO underperform RSPO counterparts under matched data size and epoch settings.

Limitations

  • Dense process reward model must be trained on collected trajectories before RSPO can run, adding implementation overhead and a dependency on a separate reward model (Llama-3.2-3B used here).
  • The quality of diverse trajectories depends on the dense reward model's coverage; a poor dense reward may still bias exploration even if outcome rewards are used for final updates.
  • RSPO gains diminish with larger (7B) models that already self-generate diverse successful trajectories, suggesting limited headroom for capable base models.
  • Off-policy ratio \(\alpha = 1/8\) and inner loop step \(k = 3\) are not extensively ablated across environments; generalization of these hyperparameters is unverified.
  • Evaluated on only two benchmarks (WebShop and ALFWorld); both are text-based and relatively constrained environments.
  • SPEAR comparison uses reproduced results under reduced compute (training size 16, 150 epochs vs. original 32, 350), which may not reflect SPEAR's true competitive upper bound.

Relevance to Agentic AI / LLM Agents

RSPO directly targets a core bottleneck in training LLM agents on long-horizon interactive tasks: the exploration problem under sparse outcome rewards. The reward-swap paradigm is a practically deployable technique that can be layered on top of any proximal policy gradient algorithm (GRPO, PPO, GiGPO), making it broadly applicable to ongoing work on agentic RL. The explicit demonstration of reward hacking when training purely with dense process rewards is a cautionary finding relevant to efforts that use learned reward models as surrogates (e.g., PRM-guided agent training). The algorithm-agnostic design means it can compose with orthogonal improvements in credit assignment (GiGPO's turn-level advantages) or curriculum strategies (SPEAR), positioning it as a modular component in multi-turn agent training pipelines.