Skip to content

Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

🕒 Published (v1): 2026-07-02 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Training a single transformer layer with RL can match—and sometimes exceed—full-parameter RL post-training. RL gains are highly concentrated in a small, structurally consistent subset of middle layers across model families, scales, and algorithms. This finding enables simple layer-aware strategies that outperform standard full-parameter RL.

Problem

RL post-training (RLVR) uniformly updates all transformer layers, implicitly assuming equal contribution across depth. Whether RL adaptation is uniformly distributed or structurally concentrated is unknown, limiting both mechanistic understanding and algorithmic efficiency of RL post-training.

Method

The authors introduce layer contribution \(C(k)\), defined as:

\[C(k) = \frac{S_k - S_{\text{base}}}{S_{\text{full}} - S_{\text{base}}}\]

where \(S_k\) is the score after training only layer \(k\) (all others frozen), \(S_{\text{base}}\) is the pretrained baseline, and \(S_{\text{full}}\) is the full-parameter RL result. For each of 7 models (1.5B–8B params), every layer is independently trained via GRPO/GiGPO/Dr. GRPO while remaining layers are frozen; gradients still flow through the full network via backpropagation, but parameter updates are restricted to layer \(k\). Layer rankings are then used to derive layer-aware training strategies: either boosting the learning rate of high-contribution layers or selectively training only the top-ranked layers. An ensemble approach applies majority voting across models trained on different individual layers.

Key Contributions

  • Introduces the layer contribution metric and a systematic single-layer RL training framework to isolate per-layer RL adaptation.
  • Establishes that RL gains are highly non-uniform: the best single layer recovers up to 114% of full-parameter gains, while the weakest recovers <30% (or even degrades performance, \(C = -0.51\) for Layer 0 of Qwen3-8B).
  • Demonstrates a universal structural pattern: high-contribution layers concentrate in the 40%–60% depth range of the transformer stack, consistently across two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), multiple datasets, and both math and agentic task domains.
  • Shows layer contribution rankings are intrinsic to pretrained weights: Spearman \(\rho = 0.76\) across two math datasets; \(\rho = 0.59\) across math vs. code tasks.
  • Develops layer-aware training strategies that outperform full-parameter RL, and shows layer-specialized model ensembles provide further complementary gains.

Results

  • Qwen3-1.7B: Best single layer (Layer 10, \(C = 1.14\)) reaches 51.8% math avg vs. 50.8% for full RL (base: 44.1%); 5/28 layers exceed \(C = 1.0\).
  • Qwen3-4B: Best single layer (Layer 16, \(C = 1.06\)) reaches 64.3% vs. 63.7% full RL (base: 52.2%).
  • Qwen3-8B: Best single layer (Layer 16, \(C = 1.07\)) reaches 67.1% vs. 66.4% full RL (base: 58.0%).
  • Best Guided (layer-aware) strategies over Qwen3 math benchmarks: +32%, +27%, +43% additional gain beyond full RL on 1.7B, 4B, 8B respectively (relative to total RL gain).
  • Qwen3-8B Best Guided achieves 69.1% math avg vs. 66.4% full RL.
  • Out-of-distribution overall contribution \(C_{\text{all}}\) closely tracks in-domain math contribution (Pearson \(r > 0.6\) on all three Qwen3 scales), indicating no overfitting.
  • Agentic tasks (ALFWorld, Qwen2.5-1.5B-Instruct and 3B-Instruct with GiGPO): same mid-layer concentration pattern observed.

Limitations

  • Learning rate is tuned for full-parameter training and reused for single-layer runs; layer-specific LR tuning might further alter rankings (though an ablation in Appendix A.7 shows rankings are robust to LR).
  • Partial layer scans for some models (Qwen2.5-1.5B-Instruct, 3B-Instruct, DeepSeek-7B) rather than exhaustive sweeps over all layers.
  • Study covers only 1.5B–8B parameter models; whether the same middle-layer pattern holds for larger models (70B+) is untested.
  • The mechanism explaining why middle layers are high-contribution is not identified—the paper is descriptive rather than causal.
  • Single-layer training may not capture interactions between layers that jointly contribute to RL gains (i.e., layer contributions are not necessarily additive).

Relevance to Agentic AI / LLM Agents

The paper directly tests agentic task adaptation (ALFWorld benchmark with Qwen2.5-Instruct models) and finds the same mid-layer concentration pattern holds for agentic RL as for mathematical reasoning, suggesting the structural finding generalizes beyond narrow reasoning tasks. For practitioners training LLM agents with RL (e.g., RLVR-based tool-use or navigation agents), the layer-aware strategy offers a practical path to stronger agents with lower compute by focusing updates on high-contribution middle layers. The consistency of layer rankings across tasks also implies that profiling on one task (e.g., math) could inform efficient RL adaptation for a different agentic domain—a potentially significant finding for multi-task agent training pipelines. More broadly, this challenges the standard full-parameter RL paradigm used in nearly all contemporary LLM agent post-training systems.