Skip to content

Why Does Feedback-Augmented Self-Distillation Fail to Improve Retrieval-Interleaved Search Agents?

🕒 Published (v1): 2026-07-20 05:07 UTC · Source: Arxiv · link

Why this paper was selected

Negative result on OPSD for retrieval-interleaved agents; tells builders what training strategies fail

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper investigates why Feedback-Augmented Self-Distillation (FA-SD) — using successful rollouts as privileged demonstrations to train a self-teacher — fails to reliably improve retrieval-interleaved search agents across seven open-domain QA benchmarks. The authors identify a novel failure mode called decoding collapse, where the model relies on recurring reasoning-and-search output templates that are agnostic to the input question, rendering the KL-based distillation signal uninformative. An EMA-stabilized self-teacher partially mitigates the collapse but requires a warm-up phase during which performance temporarily regresses.

Problem

On-policy self-distillation (OPSD) is appealing for agentic LLM training because it avoids the need for a separate stronger teacher model and can exploit within-rollout-group variance (failed vs. successful trajectories). However, whether OPSD generalizes to retrieval-interleaved search agents is unknown: unlike standard text generation, the agent's search actions determine which observations are retrieved from the environment, so token-level KL matching between a feedback-conditioned teacher and an unconditioned student must transfer long-horizon search decisions — a structurally harder problem. The paper addresses this gap empirically and mechanistically.

Method

The authors instantiate Feedback-Augmented Self-Distillation (FA-SD) in the following steps:

  1. Rollout setup. Policy \(\pi_\theta\) generates multi-step trajectories \(\tau = (r_1, s_1, o_1, \ldots, r_K, s_K, o_K, a)\) where \(r_k\) is reasoning, \(s_k\) is a search query, \(o_k = \mathcal{R}(s_k)\) is the retrieved observation, and \(a\) is the final answer.

  2. Self-teacher construction. For each question \(x\), a group of \(G\) trajectories is sampled. A correct trajectory is randomly selected as feedback \(f\) and appended to the prompt, yielding a self-teacher \(\pi_\theta^{f,t} = \pi_\theta(\cdot \mid x, \hat{y}_{<t}, f; \mathcal{R})\).

  3. FA-SD loss. The unconditioned student \(\pi_\theta^t = \pi_\theta(\cdot \mid x, \hat{y}_{<t}; \mathcal{R})\) is trained to minimize divergence from the stopgrad'd self-teacher: $\(\mathcal{L}_{\text{FA-SD}}(\theta) = \mathbb{E}_{x, \hat{y} \sim \pi_\theta(\cdot|x;\mathcal{R})} \left[ \frac{1}{|\hat{y}|} \sum_{t=1}^{|\hat{y}|} D\!\left(\pi_\theta^t \;\Big\|\; \text{stopgrad}(\pi_\theta^{f,t})\right) \right]\)$ Retrieved observations are masked; loss is computed only on assistant-generated tokens.

  4. Stabilization via EMA teacher. To counter teacher drift (model inconsistency), the teacher parameters are updated as \(\theta_t^{\text{EMA}} = \alpha \theta_{t-1}^{\text{EMA}} + (1-\alpha)\theta_t\) with \(\alpha = 0.95\), providing a slowly moving but adaptive supervision target.

  5. FA-MOPD baseline. Feedback augmentation is also applied to an external teacher (Qwen2.5-7B-Instruct) in the Multi-Rollout OPD (MOPD) framework to test whether the trick generalizes beyond self-distillation.

Experiments use Qwen2.5-3B-Base as the student, Wikipedia 2018 + E5 retriever (top-3 passages), and train/evaluate on NQ + HotpotQA (train) and seven benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle), reporting Exact Match (EM).

Key Contributions

  • FA-SD instantiation. A concrete self-distillation algorithm for retrieval-interleaved agents that uses successful within-group rollouts as privileged demonstrations.
  • Decoding collapse characterization. Identification and description of a failure mode where the model produces input-question-agnostic trajectories via recurring output templates, making the KL signal uninformative despite surface-level diversity — undetectable by standard aggregate metrics.
  • Inconsistency decomposition. Decomposing FA-SD instability into model inconsistency (teacher drift from shared updating weights) and prompt inconsistency (variance in feedback-conditioned teacher behavior across prompts).
  • EMA teacher stabilization. Demonstration that EMA-regularized FA-SD recovers from collapse and improves over the base Qwen2.5-3B-Base (avg. EM 0.206 vs. 0.114 baseline), though with a warm-up regression phase.
  • Negative result for FA-MOPD. Feedback augmentation applied to an already-strong external teacher (FA-MOPD) underperforms standard MOPD, showing prompt inconsistency is not unique to self-teachers.

Results

  • Unclipped FA-SD and PPO-clipped FA-SD: Both fail to sustain improvement; pass rates remain near zero for most of training, with only a transient rebound around step 180 before returning to near-zero.
  • Self-teacher advantage not internalized: The feedback-augmented self-teacher branch is often stronger than the unconditioned student at inference time, but the student does not reliably close that gap — ruling out a weak-teacher explanation.
  • Effective distillation signal: The fraction of rollouts providing a valid FA-SD signal decreases over training (closely tracks the pass rate), indicating the distillation bottleneck is signal sparsity, not teacher capability.
  • EMA teacher: EMA-regularized FA-SD achieves avg. EM 0.206 across seven benchmarks vs. 0.114 for Qwen2.5-3B-Base — substantial improvement, but with a temporary regression during warm-up.
  • Fixed reference teacher: Also stabilizes training and improves over baseline, but converges lower than EMA.
  • FA-MOPD vs. MOPD: FA-MOPD underperforms standard MOPD; adding privileged feedback to a strong external teacher degrades performance, implicating prompt inconsistency as the mechanism.
  • FA-SD + GRPO joint: Combined objective degrades both training pass rate and validation performance relative to GRPO alone.

Limitations

  • All experiments use a single student backbone (Qwen2.5-3B-Base); generalization to larger or instruction-tuned models is not evaluated.
  • The Wikipedia 2018 corpus and E5 retriever are fixed; results may not transfer to other retrieval stacks or live-web settings.
  • EMA requires a warm-up phase with temporary performance regression; the paper does not fully characterize when or why this warm-up stabilizes.
  • The paper is truncated — the FA-MOPD experiment discussion and several ablations (Appendix C) are referenced but not fully reproduced in the provided text.
  • Decoding collapse is identified empirically; a theoretical characterization of when it emerges and how to predict it a priori is absent.
  • The self-teacher advantage gap (Figure 2) is documented but the source of the internalization failure beyond "prompt inconsistency" is not fully resolved.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a core training challenge for LLM-based search agents: how to provide dense, stable supervision when the agent's own actions shape the observations it receives, breaking the i.i.d. assumptions that underlie standard self-distillation. The identified decoding collapse failure mode — where behavioral diversity is illusory and trajectories are input-agnostic — is a practically important diagnostic that existing aggregate metrics (EM, pass rate) can miss, making it broadly relevant to anyone training or evaluating agentic RL systems. The decomposition into model inconsistency vs. prompt inconsistency offers a principled vocabulary for understanding why richer, feedback-augmented training signals can backfire, complementing concurrent negative results on model-generated feedback in other agent settings. The EMA stabilization finding parallels BYOL-style momentum networks in representation learning, suggesting cross-paradigm solutions for self-supervised agentic training.