Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning¶
🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link
Why this paper was selected
Expert trajectory supervision enables RL for small models on complex tasks
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
TL;DR¶
Supervised Reinforcement Learning (SRL) bridges the gap between SFT (which overfits) and RLVR (which fails when pass@k ≈ 0) by decomposing expert trajectories into step-wise actions and rewarding a small LLM for action similarity at each step. The model generates a free-form internal monologue before committing to each action, receiving dense sequence-similarity rewards via GRPO. SRL outperforms both SFT and RLVR on competition-math benchmarks and generalizes to agentic software engineering tasks.
Problem¶
Small open-source LLMs (~7B parameters) cannot learn from hard reasoning problems via standard methods: RLVR fails when the policy's pass@k rate is effectively zero (no positive reward signal exists), while SFT enforces rigid token-by-token imitation of long expert demonstrations, causing overfitting and shallow generalization. This leaves a critical training gap for difficult, multi-step problems where neither approach yields meaningful improvement.
Method¶
SRL reformulates problem-solving as a sequential decision-making process over an expert trajectory \(y = \{y_{\text{step}_n}\}_{n=1}^N\).
Data construction: From a single \(N\)-step expert solution, \(N-1\) partial-trajectory training instances are created. Each instance \(x_{\text{step}_k} = [x, y_{\text{step}_1}, \ldots, y_{\text{step}_{k-1}}]\) asks the model to predict the next step \(y_{\text{step}_k}\).
Training: The policy generates a free-form inner monologue \(y'_{\text{think}}\) (inside <think> tags) followed by a predicted action \(y'_{\text{step}_k}\). A dense similarity reward is computed only on the action (not the monologue):
using Python's difflib.SequenceMatcher. A format-violation penalty of \(-1\) is applied otherwise. The policy is optimized with GRPO (Equation 1). Dynamic sampling filters batches where rollout reward standard deviation falls below threshold \(\epsilon\), ensuring non-trivial learning signal.
Sequencing: SRL can be followed by standard RLVR (outcome-reward) for further refinement.
Key Contributions¶
- SRL framework: step-wise decomposition of expert trajectories with sequence-similarity rewards via GRPO, enabling learning on \(D_{\text{hard}}\) where SFT and RLVR both fail
- Dense intrinsic reward at each reasoning step, decoupled from final-answer correctness — effective even when all rollouts produce wrong final answers
- Demonstration that multi-step guidance granularity is the key driver of performance (holistic one-step similarity reward is inferior to step-wise)
- SRL → RLVR pipeline as a cold-start strategy that achieves the strongest overall math reasoning performance
- Generalization of SRL to agentic software engineering benchmarks beyond math
Results¶
Math reasoning (Qwen2.5-7B-Instruct, trained on 1k s1K-1.1 examples):
- SRL vs. RLVR: +3.0% average across AMC23/AIME24/AIME25/Minerva (27.6 vs. 24.5)
- SRL → RLVR vs. SRL: +0.7% additional average (28.3 vs. 27.6)
- SRL vs. SFT (R1 reasoning): +11.0% average (27.6 vs. 16.6); SFT degrades below base model on most benchmarks
- SRL on AIME24: 16.7% (Avg@32) vs. R3 13.3%, vs. RLVR 11.1%
- SRL → RLVR on AIME24 greedy: 20.0% vs. base model 6.7%
- Dynamic sampling ablation: +2.9% average (27.6 vs. 24.7 without DS)
- Granularity ablation: multi-step SRL (27.6) > one-step sequence-similarity (25.9) > one-step RLVR (24.5)
- Qwen2.5-3B: +3.1% average over base model, surpassing RLVR (results partially truncated)
Limitations¶
- Requires expert trajectories with structured, numbered step decompositions; data points not adhering to this format are excluded
- Step extraction relies on teacher-generated solutions having parseable logical structure (e.g., DeepSeek R1 outline format)
- Only demonstrated on 3B and 7B parameter models; scaling behavior to larger models is not reported
- Sequence similarity reward (difflib ratio) is a lexical/surface-level metric that may not capture semantic equivalence of mathematically correct but differently expressed steps
- Training dataset is 1k examples; behavior on larger or more heterogeneous datasets is unexamined
- Agentic software engineering results are mentioned but the paper text is truncated before those numbers appear
Relevance to Agentic AI / LLM Agents¶
SRL directly addresses the training challenge for agents that must execute multi-step action sequences: its step-wise formulation — where each "action" can represent a repository command, tool call, or reasoning step — maps naturally onto agentic pipelines. The decoupling of internal monologue (flexible reasoning) from the committed action (evaluated against expert) mirrors the think-then-act structure common in LLM agent frameworks. The demonstrated generalization to software engineering tasks positions SRL as a practical training recipe for small, deployable agents that must learn from expert demonstrations when outcome-based RL signals are too sparse. For agentic AI researchers, the SRL → RLVR pipeline offers a principled cold-start strategy that may resolve the sample-efficiency bottleneck that limits fine-tuning small agents on long-horizon tasks.