TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning¶

🕒 Published (v1): 2026-06-30 17:48 UTC · Source: Arxiv · link

Why this paper was selected

Credit assignment for agentic RL; fundamental GRPO limitation with sparse outcome rewards

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

TRIAGE addresses the credit-assignment problem in agentic RL by adding a semantic role axis to GRPO's outcome-only advantage signal. An LLM judge classifies each environment-facing segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and fixed role-conditioned process rewards correct the two structural blind spots of uniform outcome credit. On ALFWorld, Search-QA, and WebShop, TRIAGE consistently improves success rates over GRPO and outperforms scalar process-reward and value-baseline alternatives.

Problem¶

Standard GRPO broadcasts a single trajectory-level advantage to every token uniformly. In agentic settings this creates two systematic errors: (1) useful exploratory segments in failed rollouts are over-punished, discouraging information-gathering behavior in partially observable tasks; (2) regressive segments (wrong edits, wrong purchases, redundant clicks) in successful rollouts receive unwarranted positive credit because the agent later recovered. Neither outcome-statistical estimators nor generic process scorers resolve these "conflict cells" because they assign role-agnostic scores rather than distinguishing exploration from no-progress, or regression from decisive action.

Method¶

TRIAGE adds a structured role-judge layer on top of standard GRPO without replacing the verifier or learning a value model.

Role taxonomy. Each environment-facing segment $a_{i,k}$ is assigned $\rho_{i,k} \in \{D, E, N, R\}$: Decisive (verifier-checkable task-state progress), Exploration (information-state progress without subgoal completion), No-progress (harmless, changes neither state), Regression (corrupts state or repeats without information gain).

Judge. A Qwen3-8B-thinking LLM judge receives a bounded local context window (up to five previous and five future action–observation pairs, without the final verifier outcome) and outputs one role label plus auxiliary evidence strings. The evidence requirement is a calibration knob that aids the hard cell (R inside successful rollouts).

Role-conditioned advantage. The segment advantage is: $$A_{i,k}^{\text{TRIAGE}} = A_i^{\text{GRPO}} + \lambda \, c_{\hat{\rho}_{i,k}}$$ with fixed role constants $(c_D, c_E, c_N, c_R) = (1, 0.5, -0.1, -0.5)$ and $\lambda \in \{0.2, 0.4\}$ selected per environment on the training split. Segment advantages are batch-whitened before being broadcast to segment tokens; prompt and observation tokens are excluded from the policy loss. No judge is used at evaluation time.

Theoretical grounding. Define the credit residual $\delta_{i,k} \triangleq A^*_{i,k} - A_i^{\text{GRPO}}$. Among all corrections measurable from $\rho_{i,k}$, the MSE-optimal correction is $g^*(\rho) = \mathbb{E}[\delta_{i,k} \mid \rho_{i,k} = \rho]$. The MSE reduction is $\mathbb{E}[\mathbb{E}[\delta_{i,k} \mid \rho_{i,k}]^2] \geq 0$, and for the fixed constants the MSE change relative to GRPO is $\Delta\text{MSE} = \lambda^2 \mathbb{E}[c_{\hat\rho}^2] - 2\lambda \,\text{Cov}(c_{\hat\rho}, \delta)$, which is negative (improvement) for sufficiently small $\lambda$ whenever $c_\rho$ is positively aligned with $g^*(\rho)$.

Key Contributions¶

Identifies two structural blind spots of outcome-only segment credit—useful exploration in failed rollouts and regression inside successful rollouts—and defines a four-role taxonomy ($D, E, N, R$) as a role axis complementing the outcome axis.
Introduces TRIAGE: a structured LLM judge for semantic role typing combined with fixed role-conditioned process rewards added to the GRPO advantage.
Theoretical proof that role-conditioned credit is the MSE-optimal segment correction expressible from role labels alone (Proposition 1), connecting fixed role constants to lower-variance policy gradients.
Empirical disentanglement showing that judge reliability on the hard R-in-success cell (not mere addition of dense rewards) drives the gains, via ablations comparing thinking vs. no-think judges and evidence vs. no-evidence prompts.

Results¶

ALFWorld (Qwen2.5-7B): GRPO 79.6 ± 1.9% → TRIAGE 87.5 ± 2.4%; ties GiGPO (87.8 ± 2.2%) and beats shared-backbone value baseline (85.2 ± 2.7%).
WebShop (Qwen2.5-7B): GRPO 70.1 ± 2.3% → TRIAGE 77.2 ± 2.2%; beats GiGPO (74.3 ± 2.9%) and PPO (71.5 ± 2.0%).
Search-QA (Qwen2.5-7B): GRPO 43.3% → TRIAGE 48.1% (single run); GiGPO inapplicable.
Qwen3-1.7B gains: ALFWorld 45.2 → 56.4%; WebShop 37.5 → 55.9%; Search-QA 39.4 → 42.3%.
No-think judge degrades TRIAGE below GRPO on ALFWorld and WebShop for both models, confirming that gains are from accurate role typing, not from extra reward density.
Efficiency: On completed rollouts, TRIAGE reduces environment-facing turns by 10.4% (ALFWorld) and 14.8% (WebShop) relative to GRPO.
Ablations: Removing the regression penalty ($c_R = 0$) drops ALFWorld to 81.4%; removing the exploration bonus ($c_E = 0$) drops to 85.8%, confirming regression suppression as the dominant contributor and exploration credit as a consistent secondary gain.
Judge F1 (135 labeled segments): R-in-success: 8B-think 86.1 vs. 8B-no-think 29.2; E-in-failure: 8B-think 95.2 vs. 8B-no-think 90.0, showing thinking is specifically critical for the hard conflict cell.

Limitations¶

Judge reliability is load-bearing: a weaker (no-think) judge inverts performance below the GRPO baseline, making TRIAGE sensitive to the judge's ability to detect regression inside successful rollouts.
$\lambda$ requires per-environment tuning on the training split; the role constants are fixed by design but may not be well-calibrated for domains with different action distributions.
Search-QA results are single-run due to the cost of multi-turn retrieval rollouts, so variance is unquantified for that benchmark.
The theoretical guarantee (Proposition 1) requires that the judge's labels are aligned with $g^*(\rho)$; the paper explicitly flags failure modes in Appendix B but does not derive a sample-complexity bound on judge accuracy needed for guaranteed improvement.
Evaluation is limited to three benchmarks in relatively constrained action spaces (ALFWorld templated commands, WebShop search/click, Search-QA query/answer); generalization to open-ended tool-use or code-editing agents is not demonstrated.
The per-segment LLM judge call adds training-time inference cost (though no judge is needed at evaluation).

Relevance to Agentic AI / LLM Agents¶

TRIAGE directly addresses the core challenge of training LLM agents with RL when reward is sparse and delayed: how to assign meaningful credit to individual tool calls, searches, or navigation commands within a multi-step trajectory. The role-typed framework formalizes the intuition that exploration and regression are semantically distinct from mere progress, providing a principled way to preserve information-gathering behavior—a known fragility point in agentic RL where sparse-reward training tends to suppress exploration. The result that a structured LLM judge (rather than a learned value function) can serve as a reliable credit-assigner connects to the broader trend of using LLMs as process-level supervisors in agent training pipelines. For researchers building RL-trained tool-use or web agents, TRIAGE offers a practical, modular augmentation to GRPO-style training that improves both task success and trajectory efficiency without requiring a separate value network.