Skip to content

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

🕒 Published (v1): 2026-06-25 09:24 UTC · Source: Arxiv · link

Why this paper was selected

On-policy skill distillation adds dense signal to sparse outcome-based RL for language agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OPID augments outcome-based RL for LLM agents by extracting hierarchical "skills" (episode-level workflows and step-level critical-decision rules) directly from completed on-policy trajectories, then distilling them as dense token-level advantages without requiring external skill libraries. The key insight is that hindsight skills derived from the policy's own rollouts are distribution-matched by construction, sidestepping the state-drift problem of retrieved off-policy skills. OPID consistently outperforms both outcome-only RL (GRPO) and skill-distillation baselines across three multi-step agentic benchmarks.

Problem

Outcome-based agentic RL (e.g., GRPO) provides only sparse, terminal rewards: a binary success signal reveals whether a trajectory succeeded but gives no credit assignment to individual decisions. Existing skill-conditioned self-distillation methods address this by injecting privileged natural-language skills, but rely on external skill libraries or retrieved memories—introducing maintenance overhead and distribution mismatch when retrieved skills diverge from the current policy's state distribution, especially severe in long-horizon interaction where small deviations cause state drift.

Method

OPID operates in three stages per training iteration:

  1. On-policy skill extraction. After collecting a group of \(N\) on-policy trajectories \(\{Ï„^{(1)}, \ldots, Ï„^{(N)}\}\), an LLM-based analyzer \(\mathcal{A}\) processes each complete trajectory and outputs: (a) an episode-level skill \(s^\text{ep}_Ï„\) summarizing the global workflow (success) or failure-avoidance rule (failure), and (b) a sparse set of step-level skills \(\{s^\text{step}_{Ï„,t}\}_{t \in \mathcal{C}_Ï„}\) for critical timesteps \(\mathcal{C}_Ï„\) identified by the analyzer.

  2. Critical-first routing and skill-conditioned scoring. For each timestep \(t\), the routed skill is: $\(s_{Ï„,t} = \begin{cases} s^\text{step}_{Ï„,t} & \text{if } t \in \mathcal{C}_Ï„ \\ s^\text{ep}_Ï„ & \text{otherwise.} \end{cases}\)$ The selected skill is injected into the interaction history to form \(\tilde{h}_{Ï„,t} = \mathcal{H}(h_{Ï„,t}, s_{Ï„,t})\). The old policy \(\pi_{\theta_\text{old}}\) re-scores the same sampled response \(y_{Ï„,t}\) under both the original and skill-augmented histories. The token-level skill advantage is: $\(A^\text{skill}_{Ï„,t,\ell} = \bigl(\log \pi_{\theta_\text{old}}(y_{Ï„,t,\ell} \mid \tilde{h}_{Ï„,t}) - \log \pi_{\theta_\text{old}}(y_{Ï„,t,\ell} \mid h_{Ï„,t})\bigr) \cdot m_{Ï„,t,\ell}\)$ where \(m_{Ï„,t,\ell}\) is the valid-token mask. No response is regenerated.

  3. Combined advantage optimization. The GRPO group-relative episode advantage \(A^\text{ep}_Ï„ = (R(Ï„) - \mu_q)/\sigma_q\) is broadcast token-wise and combined with the skill advantage: $\(A^\text{OPID}_{Ï„,t,\ell} = A^\text{ep}_{Ï„,t,\ell} + \lambda_\text{skill} A^\text{skill}_{Ï„,t,\ell}\)$ This is plugged into the standard clipped PPO-style objective plus a KL penalty. At inference time, no analyzer, skill retrieval, or augmented context is used.

Key Contributions

  • On-policy hindsight skill extraction: skills derived from the current policy's own completed trajectories, eliminating off-policy retrieval and distribution mismatch.
  • Hierarchical skill representation: two-level abstraction—episode-level (global workflow/failure rule) and step-level (local critical decision)—covering different granularities of long-horizon decisions.
  • Critical-first routing: a deterministic mechanism that applies the more precise step-level skill only at identified critical timesteps and falls back to episode-level elsewhere.
  • Dense token-level shaping within outcome-RL: skill advantage adds fine-grained credit assignment without replacing the primary RL objective; the combined advantage is directly integrated into a GRPO-style clipped objective.
  • Zero inference overhead: the skill extraction and conditioning are training-only; the learned policy requires no special context at test time.

Results

All figures are success rate (%) on ALFWorld, accuracy on Search-based QA, and score/success rate on WebShop. Baselines include Vanilla, Skill-Prompt, GRPO, Skill-GRPO, OPSD, GRPO+OPSD, Skill-SD, RLSD, and SDAR.

Qwen2.5-3B vs. GRPO baseline: - ALFWorld: 84.3 vs. 75.0 (+9.3 pts) - Search-based QA: 45.0 vs. 36.4 (+8.6 pts) - WebShop (success): 74.2 vs. 63.3 (+10.9 pts)

Qwen2.5-7B vs. GRPO: - ALFWorld: 90.0 vs. 81.2 (+8.8 pts), best among all methods on this backbone - Search-based QA: 49.2 vs. 42.0 (+7.2 pts) - WebShop (success): 79.7 vs. 72.6 (+7.1 pts)

Qwen3-1.7B vs. GRPO: - ALFWorld: 58.9 vs. 46.1 (+12.8 pts), best of all methods - WebShop (success): 64.8 vs. 38.3 (+26.5 pts), strongest gain across the entire table - Search-based QA: 40.4 vs. 40.8 (flat; OPID does not help on this model–domain pair)

vs. strongest skill/distillation baselines: - ALFWorld (7B): +1.7 pts over best baseline (90.0 vs. 88.3 SDAR) - ALFWorld (1.7B): +5.0 pts over best baseline (58.9 vs. 53.9 SDAR) - WebShop (1.7B): +6.2 pts over best baseline success rate (64.8 vs. 58.6 SDAR)

Limitations

  • The LLM-based analyzer \(\mathcal{A}\) is invoked every training iteration per trajectory, adding compute cost during training (though zero cost at inference).
  • Skill quality depends on the analyzer's ability to correctly identify critical timesteps and generate accurate hindsight explanations; poor analyzer outputs could introduce noisy supervision.
  • Search-based QA improvements are modest or absent on smaller backbones (Qwen3-1.7B), suggesting OPID's benefit is domain- and model-scale-dependent.
  • No ablation isolating the contribution of the analyzer's critical-step identification vs. the hierarchical skill structure is presented in the excerpted text.
  • Evaluated only on three benchmark families; generalization to code agents, tool-use agents, or web-browsing agents is not demonstrated.

Relevance to Agentic AI / LLM Agents

OPID directly addresses the credit-assignment bottleneck in RL-trained LLM agents—one of the most active open problems as GRPO-style training becomes the standard post-training paradigm for multi-step agents. By showing that completed rollouts from the current policy are a sufficient and distribution-correct source of dense supervision, the work removes the need for external skill libraries (a significant engineering burden), making skill-augmented RL more self-contained and deployable. The hierarchical routing idea (episode-level vs. step-level, triggered by criticality) is a generalizable design principle for any long-horizon agent where different decisions warrant different supervision granularity. This connects closely to the broader SDAR/RLSD line of work on self-distillation for agents, and represents a step toward agents that improve their own procedural knowledge purely from their own experience.