Skip to content

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

🕒 Published (v1): 2026-07-16 09:57 UTC · Source: Arxiv · link

Why this paper was selected

Self-evolving on-policy distillation closes RL sample-efficiency gap; directly improves agentic training recipes

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SEED (Self-Evolving On-Policy Distillation) addresses the sparse-reward problem in agentic RL by converting completed on-policy trajectories into natural-language "hindsight skills" and distilling their behavioral effect back into the policy as dense token-level supervision. The same model serves as both the RL actor and the trajectory analyzer, so as the policy improves, the quality of hindsight supervision co-evolves. At inference time, no external memory, skill bank, or augmented prompt is required.

Problem

Outcome-based agentic RL assigns rewards only at the trajectory level, providing no signal about which intermediate observations, actions, or tool calls should be reinforced or corrected. This supervision gap between episode-level outcomes and token-level policy learning is especially acute in long-horizon tasks where a failed trajectory may contain many locally useful behaviors and a successful one may contain reusable strategies that a scalar reward never identifies. Prior hindsight approaches (verbal reflection, episodic memory, skill distillation) treat hindsight as static experience or inference-time context, and their supervision becomes stale as the policy evolves.

Method

SEED has two training stages.

Stage 1 — Hindsight Skill SFT. An offline trajectory pool (\(M=180\) tasks, \(K_0=8\) rollouts each) is collected with the base policy. An external analyzer (GLM-5.2) annotates each trajectory with a natural-language hindsight skill \(s_\tau\) encoding reusable strategies (for successes) or failure-avoidance rules (for failures). The base model is fine-tuned for 3 epochs on valid \((x_\tau, s_\tau)\) pairs via standard negative log-likelihood.

Stage 2 — Self-Evolving On-Policy Distillation. The SFT checkpoint initializes the RL policy. Each iteration: (i) the frozen snapshot \(\pi_{\theta_\text{old}}\) collects \(N=8\) rollouts per task; (ii) the same snapshot, acting as the analyzer, extracts a hindsight skill \(s_q^{(n)}\) from each completed trajectory; (iii) each sampled action \(a_{q,n,t}\) is re-scored under two contexts — the ordinary history \(h_{q,n,t}\) (student branch) and the skill-augmented history \(\tilde{h}_{q,n,t} = H(h_{q,n,t}, s_q^{(n)})\) (teacher branch); (iv) the skill-induced log-probability shift \(\Delta_{q,n,t,\ell} = \text{sg}[\ell^\text{skill}_{q,n,t,\ell} - \ell^\theta_{q,n,t,\ell}]\) is mapped through a confidence gate \(g = \sigma(\beta_\text{opd} \Delta)\), yielding the OPD loss:

\[\mathcal{L}_\text{OPD}(\theta) = \mathbb{E}_{q,n,t,\ell}\!\left[m_{q,n,t,\ell} \cdot g_{q,n,t,\ell} \cdot \text{sg}\!\left[\ell^\text{skill}_{q,n,t,\ell}\right] - \ell^\theta_{q,n,t,\ell}\right]\]

This is gradient-equivalent to gate-weighted negative log-likelihood on the student branch only; the joint objective is \(\mathcal{L}_\text{SEED} = \mathcal{L}_\text{RL} + \lambda_\text{opd}\mathcal{L}_\text{OPD}\), where \(\mathcal{L}_\text{RL}\) is clipped GRPO with KL regularization. Refreshing \(\theta_\text{old}\) after each update closes the self-evolving loop.

Key Contributions

  • Self-evolving loop: the same policy checkpoint is simultaneously the rollout actor and the hindsight-skill analyzer, so decision-making and skill-analysis capabilities co-improve.
  • Policy-synchronized OPD mechanism: skill-induced log-probability shifts on sampled actions provide dense token-level supervision without any external teacher or static skill dataset.
  • Zero inference-time overhead: all hindsight skills are training-time only; deployed agents use the ordinary interaction history with no analyzer, skill bank, or retrieval module.
  • Empirical validation across three diverse long-horizon agentic benchmarks (embodied control, web navigation, search-based QA) with multiple backbone sizes.

Results

All metrics are success rate (%) or accuracy (%) unless noted; higher is better.

  • ALFWorld (macro-avg over 6 task families): SEED 91.8 vs. SDAR 84.4, RLSD 79.7, Skill-SD 73.4, GRPO 60.2, Vanilla 21.9.
  • WebShop (success rate): SEED 78.9 vs. SDAR 68.0, RLSD 66.4, Skill-SD 64.0, GRPO 60.9, Vanilla 0.8.
  • Search-based QA (macro-avg over 7 datasets): SEED 45.7 vs. SDAR 43.4, RLSD 43.8, Skill-SD 44.1, GRPO 34.1, Vanilla 23.9.
  • WebShop task-completion score: SEED 88.5 vs. SDAR 85.0, RLSD 84.4, Vanilla 6.7.
  • SEED outperforms Skill-Prompt* (skills given at test time) on ALFWorld (91.8 vs. 28.9) and WebShop success (78.9 vs. 36.4), showing internalization exceeds inference-time skill prompting.
  • Backbone models: Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen3-1.7B-Instruct; rollout group size \(N=8\); 150 RL updates.

Limitations

  • Stage 1 requires an external analyzer (GLM-5.2) for initial skill annotation; this introduces a dependency on a capable frontier model to bootstrap the SFT data.
  • Only 180 training tasks and 1,440 SFT trajectories are used; it is unclear how performance scales with dataset size or task diversity.
  • Benchmarks (ALFWorld, WebShop, Search-based QA) are text-based or limited-vision; generalization to richer multimodal or open-ended real-world environments is not fully demonstrated.
  • The confidence gate introduces hyperparameter \(\beta_\text{opd}\) and the loss weight \(\lambda_\text{opd}\); sensitivity analysis is deferred to appendices and may not transfer across task types.
  • The self-evolving loop doubles the forward-pass cost per update step (actor + analyzer), which is not discussed in terms of wall-clock training overhead.

Relevance to Agentic AI / LLM Agents

SEED directly targets the central bottleneck of training LLM agents with RL: sparse trajectory-level rewards provide no credit assignment at the decision level. By converting completed trajectories into self-generated, self-evolving dense supervision — with no inference-time cost — SEED advances the practical toolkit for post-training agentic LLMs on long-horizon, multi-turn tasks. The framework connects hindsight RL, on-policy self-distillation, and skill-based agent training into a unified loop, and its zero-overhead deployment property makes it compatible with any existing agentic inference stack. Researchers tracking LLM agent training, reward shaping, and distillation-based policy improvement will find SEED directly relevant as a scalable alternative to process reward models or external verifiers.