Skip to content

Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

🕒 Published (v1): 2026-06-24 14:02 UTC · Source: Arxiv · Venue: EMNLP 2026 · link

Why this paper was selected

EMNLP 2026; semantic consistency resolves step-level credit assignment in sparse-reward agentic RL

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Group-based RL methods for LLM agents assign every step in a failed trajectory the same negative credit, even when some steps are nearly identical to those in successful rollouts—a failure mode the authors call semantic credit inconsistency. SCPO is a reward-shaping plugin that recovers step-level credit for failed steps by matching them against a successful sibling in the same rollout group using a frozen cross-encoder. At 1.5B parameters, SCPO reaches 93.7% on ALFWorld and 74.8% task-success on WebShop, outperforming GiGPO by +7.0 and +9.8 points respectively.

Problem

In sparse-reward, multi-turn agentic RL, group-based methods (GRPO, RLOO, GiGPO) assign advantages based on trajectory-level outcomes. A locally correct intermediate step in a failed rollout receives the same negative signal as a genuinely wrong step, because credit is tied to the trajectory's terminal outcome. Two semantically near-identical steps—one from a successful rollout, one from a rollout that fails only later—thus receive opposite training signals, injecting conflicting gradients and discarding recoverable partial progress inside failed trajectories.

Method

SCPO is a reward-level plugin that sits between step-reward construction and advantage estimation; it wraps GiGPO without modifying the rollout collector, policy objective, or advantage estimator.

For each rollout group of \(n\) trajectories \(\{\tau_i\}_{i=1}^n\):

  1. Success reference selection. Partition trajectories by terminal outcome into successful set \(S\) and failed set \(W\). Select the longest successful trajectory \(\tau^* = \arg\max_{i \in S} |\tau_i|\) as the reference (longest maximizes reusable reference positions).

  2. Semantic similarity matrix. Encode each step as a text string from its \((s_t, a_t)\) pair. For each failed sibling \(w\), compute similarity matrix \(M \in [0,1]^{m \times \ell}\) with \(M_{u,v} = f(x^*_u, x^w_v)\) using a frozen cross-encoder (BGE-Reranker-v2-m3); \(m\) and \(\ell\) are the filtered reference and failed-trajectory lengths.

  3. Monotonic credit assignment. A failed step \(v\) earns credit at reference position \(j\) only if (i) \(M_{j,v} \geq \theta\) and (ii) \(j\) is beyond every already-credited position (strictly monotonic advance through the reference). Credit is: $\(r^{\text{SCPO}}_{w,v} = \frac{\max(0,\, M_{j,v} - \lambda)}{1 - \lambda}\)$ with soft base \(\lambda\); steps that fail either condition receive zero. This prevents repeated template matches from accumulating spurious reward.

  4. Allocation order. Rather than processing failed steps chronologically (which wastes early reference positions on low-discriminative navigation templates), SCPO uses a reordered allocation so steps from all trajectory positions compete equally for reference positions.

  5. Reward integration. Recovered credit, scaled by \(\alpha\), is added to the GiGPO step-level discounted return \(\tilde{R}^{(i)}_t = R^{(i)}_t + \alpha r^{\text{SCPO}}_{i,t}\) for failed siblings only. The episode-level advantage \(A_E(\tau_i)\) is computed from the unmodified terminal outcome, so SCPO cannot relabel a failed trajectory as successful.

Key Contributions

  • Identifies semantic credit inconsistency as a distinct failure mode in sparse-reward agentic RL: near-identical steps in successful vs. failed rollouts receive opposite training signals.
  • Proposes SCPO, a value-free reward-shaping plugin requiring no learned critic, process reward model, verifier, expert demonstrations, or extra rollouts—only one frozen cross-encoder forward pass over already-collected steps.
  • First application of frozen cross-encoder semantic step matching to agentic RL credit assignment.
  • Two design ablations: (a) reordered allocation order (+3.3 pts over chronological), (b) monotonic crediting (+3.6 pts over unrestricted re-matching).

Results

  • ALFWorld (Qwen2.5-1.5B): SCPO 93.7±4.1% vs. GiGPO 86.7±1.7% (+7.0), vs. HGPO 92.77±1.08% (prior SOTA); gains concentrated in LOOK (+21.1), COOL (+13.0), PICK2 (+14.5).
  • WebShop (Qwen2.5-1.5B): SCPO 74.8±2.0% task-success vs. GiGPO 65.0±3.2% (+9.8), vs. HGPO 71.54±4.00%.
  • ALFWorld (Qwen2.5-7B): SCPO 95.3±1.6% vs. GiGPO 90.8±1.3%, on par with HGPO 95.44±0.62%.
  • WebShop (Qwen2.5-7B): SCPO 77.5±3.0% vs. GiGPO 72.8±3.2%.
  • Gains are largest on the hardest multi-step task families (LOOK, COOL, PICK2) and smallest on near-saturated single-step families (PICK, CLEAN, HEAT), consistent with the hypothesis that partial-progress recovery matters most where failed rollouts contain recoverable useful behavior.
  • Ablation: chronological allocation order −3.3 pts; removing monotonicity −3.6 pts; shortest reference −2.0 pts.

Limitations

  • Evaluated on two benchmarks (ALFWorld, WebShop) with binary or near-binary rewards; generalization to richer reward landscapes is unverified.
  • Gains narrow significantly at 7B; the method is most impactful for weaker/smaller policies where failed rollouts still contain high-quality partial behavior.
  • Requires at least one successful sibling per rollout group; groups with zero successes receive no SCPO credit, which may be problematic in very early training or low-success regimes.
  • The frozen cross-encoder (BGE-Reranker-v2-m3) adds inference-time cost proportional to \(m \times \ell\); not profiled at scale beyond 7B.
  • Longest-reference heuristic and hyperparameters (\(\theta\), \(\lambda\), \(\alpha\), \(\omega\)) require tuning; sensitivity beyond the ablated dimensions is not reported.
  • Code not yet released (planned post-submission).

Relevance to Agentic AI / LLM Agents

SCPO directly addresses a core challenge in training LLM agents via RL: the credit assignment problem in long-horizon, sparse-reward trajectories. By recovering partial credit from failed rollouts without any learned auxiliary model, it offers a lightweight but principled improvement over the dominant group-based RL paradigm (GRPO/GiGPO), which is increasingly the standard post-training recipe for agentic LLMs. The frozen-encoder semantic matching approach generalizes naturally to any agentic RL setting where rollout groups contain mixed-outcome trajectories, making it broadly applicable to web agents, embodied agents, and tool-using agents. The identification of semantic credit inconsistency as a named failure mode also provides a useful conceptual handle for the field, connecting to open problems in process reward modeling and hindsight credit assignment.