Skip to content

PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning

🕒 Published (v1): 2026-07-23 15:24 UTC · Source: Arxiv · link

Why this paper was selected

Policy-aware scaffolding for long-horizon RL agents; addresses exploration collapse in sparse-reward settings

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PATS reframes external skills as disposable training scaffolds for LLM agent RL rather than persistent artifacts to retain or distill. A dynamic Bank of experience-derived entries expands when a weak policy fails, revises as residual errors shift, and compresses once the policy becomes competent—then is discarded entirely at deployment. On ALFWorld and WebShop, PATS improves over GRPO by up to 18.6 SR points while reducing deployment interaction tokens by ~30–39%.

Problem

In long-horizon agentic RL, a weak policy repeats stereotyped failures, causing GRPO rollout groups to have near-uniform outcomes (\(r \approx 0\) everywhere) and therefore near-zero group-relative advantage signal. Prior skill-centric methods (SkillRL, SKILL0, SLIM, SEED) treat skills as assets to retrieve, curate, attribute, or internalize—adding control objectives (e.g., leave-one-out contribution estimation, auxiliary distillation) that need not align with making the next rollout group more informative.

Method

Policy-Aware Training Scaffolding (PATS) keeps the task distribution, environment reward, and GRPO objective fixed and adapts only a structured textual scaffold Bank \(B_k\) alongside policy training.

The Bank has three semantic layers: general principles \(B_k^{\text{gen}}\), task-type procedures \(B_{k,\tau}^{\text{task}}\), and recurring-mistake entries \(B_k^{\text{mistake}}\). A frozen snapshot \(B_k^e\) conditions every rollout in iteration \(k\); edits derived from those rollouts become visible only in snapshot \(k+1\).

Evidence aggregation. Each GRPO group \(G_{q,k}\) of \(n\) parallel attempts on task \(q\) yields an evidence card containing the group success rate \(s_{q,k}\), deterministic aggregate features \(\Phi(G_{q,k}^+)\)/\(\Phi(G_{q,k}^-)\), and representative traces (observations + executable actions only, no free-form reasoning). Cards are aggregated by task type \(\tau\).

Competence-conditioned controller. An EMA success rate \(\bar{s}_{\tau,k} = \alpha s_{\tau,k} + (1-\alpha)\bar{s}_{\tau,k-1}\) and global scaffold pressure \(p_k = \max\!\left(\frac{|B_k|}{N_{\max}}, \frac{\text{Tok}(B_k)}{L_{\max}}\right)\) drive a four-mode controller:

\[\mu_{\tau,k} = \begin{cases} \text{FORCED\_PRUNE} & p_k \ge 1 \\ \text{COMPRESS} & \bar{s}_{\tau,k} \ge \rho_c \\ \text{REVISE} & \rho_r \le \bar{s}_{\tau,k} < \rho_c \\ \text{EXPAND} & \bar{s}_{\tau,k} < \rho_r \end{cases}\]

Constrained refiner. An LLM refiner proposes one of four edit operations (add principle, add mistake, update entry, delete entry). A deterministic validator enforces schema validity, evidence-support requirements (≥2 supporting cards for new entries), mode-direction constraints, and capacity budgets; only valid proposals are atomically committed.

Deployment. The scaffold, refiner, evidence builder, and controller are all discarded. The final policy runs as \(\pi_{\theta^\star}(\cdot \mid q, \emptyset)\).

Key Contributions

  • Policy-centric training paradigm: reframes skills as temporary, competence-driven scaffolds rather than persistent deployment assets.
  • Closed evidence loop: GRPO rollout groups feed back directly into scaffold adaptation without per-skill value estimation or auxiliary distillation objectives.
  • Four-mode adaptive controller: EXPAND → REVISE → COMPRESS → FORCED_PRUNE supports the full competence trajectory; REVISE in particular enables mid-training content replacement rather than monotonic growth or withdrawal.
  • Constrained deterministic validator: ensures scaffold edits are auditable and schema-compliant, preventing refiner hallucination from corrupting the Bank.
  • Scaffold-free deployment guarantee: SFT interface initialization + RL on environmental rewards means the trained policy never requires a retrieved skill library at test time.

Results

  • ALFWorld (1.5B): PATS achieves 80.71% SR vs. GRPO 67.86% (+12.9 pp); SkillRL (without test-time skills) 47.38%. Reduces interaction tokens from 14,723 to 10,245.
  • ALFWorld (7B): PATS 89.29% SR vs. GRPO 71.67% (+17.6 pp); SkillRL (no skills) 84.76%. Tokens reduced from 17,060 to 10,455.
  • WebShop (1.5B): PATS SR 56.33% vs. GRPO 52.80% (+3.5 pp); Score 0.795 vs. 0.731.
  • WebShop (7B): PATS SR 76.20% vs. GRPO 57.60% (+18.6 pp); Score 0.878 vs. 0.784.
  • Scaffold removal overhead: removing the test-time scaffold costs ≤1.66 SR points on ALFWorld and ≤0.012 WebShop Score (negligible).
  • Search-augmented QA (7 benchmarks, 7B): PATS improves over SKILL0 by +3.5 avg points; uses 32.1% fewer prompt tokens than SkillRL (which retains its skill library at inference).
  • Ablation: removing online adaptation −7.14 SR on 1.5B ALFWorld; removing REVISE alone −12.62 SR (largest single ablation drop).
  • Controlled NLL diagnostic: NLL gap between supported and unsupported teacher forcing narrows by 38.3% relative over 50 RL steps, and unsupported success rises from 17.66% to 22.97% vs. 17.66% for no-band RL.

Limitations

  • Requires a scaffold-interface SFT initialization stage before RL; warm-starting without it is insufficient.
  • The LLM-based refiner introduces additional per-iteration inference cost not accounted for in "deployment tokens."
  • The controlled skill-removal diagnostic spans only two ALFWorld game files, so evidence for skill internalization is narrow and not generalizable.
  • On search-augmented QA, SkillRL with retained skills at inference still outperforms PATS, meaning the performance–context trade-off favors PATS only when deployment context must be removed.
  • Task-type competence tracking assumes tasks can be cleanly categorized into types \(\tau\); tasks without sufficient group samples per type may not trigger meaningful scaffold updates.

Relevance to Agentic AI / LLM Agents

PATS directly addresses the sample-efficiency bottleneck in agentic RL for long-horizon tasks—the collapse of GRPO signal when a weak policy can barely succeed—by introducing an adaptive, evidence-driven training scaffold that shapes the sampling distribution without modifying the reward or optimization objective. This is complementary to and cleanly separable from policy-gradient advances (PPO, GRPO variants) and problem-side curricula, making it a drop-in component for future agentic RL pipelines. The framework's insistence on scaffold-free deployment is significant for practitioners: it shows that structured textual guidance can be used to bootstrap capable agents without creating inference-time retrieval dependencies, a recurring concern in LLM agent deployment. The four-mode controller (EXPAND/REVISE/COMPRESS/FORCED_PRUNE) operationalizes the intuition that different stages of RL training require qualitatively different kinds of external support, which connects to broader debates about when and how much scaffolding LLM agents need.