Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning¶

🕒 Published (v1): 2026-06-16 06:55 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; training-free verbal RL lets agents learn directly from world feedback

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper identifies the retention-forgetting dilemma in verbal reinforcement learning agents operating in non-stationary environments: retaining stale rules causes negative transfer, while discarding them causes catastrophic forgetting when conditions recur. The authors propose a three-layer architecture (rules, evidence, skills) with a feedback-driven critic–proposer–curator curation loop that governs which rules to trust and apply. On financial forecasting, uncurated experience degrades accuracy below zero-shot, while the full system yields +5.3pp directional accuracy and a 60% reduction in maximum drawdown.

Problem¶

Training-free verbal RL methods invest heavily in experience extraction (generating good rules from outcomes) but underinvest in insight governance (managing rules once accumulated). In non-stationary environments, existing methods satisfy at most a subset of four requirements: outcome-driven evaluation (R1), persistent structured evidence (R2), non-monotonic knowledge lifecycle (R3), and compositional governance (R4). No prior method satisfies all four simultaneously, causing accumulated experience to actively harm performance when conditions shift.

Method¶

The architecture has three layers connected by a batch-level critic–proposer–curator loop:

Rules ($L_k$): Natural-language trigger-condition/corrective-action pairs distilled from world feedback. Rules are deprecated (context-removed) but never deleted, satisfying R3.
Evidence ($\Xi_k$): Per-rule append-only logs recording each episode's outcome, whether the rule helped or hurt, and under what conditions. Evidence persists across deprecation, satisfying R1 and R2.
Skills ($S_k$): Higher-order routing policies that specify priority orderings among active rules, conflict resolution, and abstention conditions. Skills evolve as evidence accumulates, satisfying R4.

The formal update per batch $B_k = \{(x_t, a_t, y_t)\}$ is:

\[E_k = \text{CRITIC}(B_k, L_{k-1}, S_{k-1})$$ $$\Xi_k, P_k = \text{PROPOSER}(E_k, L_{k-1}, \Xi_{k-1})$$ $$L_k, S_k = \text{CURATOR}(L_{k-1}, S_{k-1}, \Xi_k, P_k)\]

The critic compares rule-augmented reasoning against a zero-shot baseline on the same observed outcome (attribution to specific rules via chain-of-thought, not just binary success). The proposer appends evaluations to evidence logs and proposes new rules for uncovered error patterns. The curator deprecates rules with consistently negative evidence and evolves skills. At inference, only $L_K^{\text{active}}$ and $S_K$ enter the agent's context; no model parameters are updated.

The case study uses Qwen3-VL-235B as the base agent on daily OHLCV candlestick charts (AAPL, AMZN, FB, GOOGL, MSFT; 2013–2016 learning, 2017 test); critic/proposer/curator roles use Claude Sonnet 4.6; batches of ~16 samples.

Key Contributions¶

Formal framing of the retention-forgetting dilemma as the central design challenge for verbal RL from non-stationary world feedback.
Four requirements (R1–R4) as a diagnostic checklist, applied to map gaps in Reflexion, ExpeL, trajectory-informed tips, and meta-MDP libraries.
Three-layer architecture with append-only evidence logs and an evolving skill layer—the first verbal RL system satisfying all four requirements simultaneously.
Empirical demonstration that governance, not quantity of accumulated experience, determines whether an agent improves or degrades.

Results¶

All baselines use the same Qwen3-VL-235B base agent on 2017 test set (5 runs, mean ± std):

Method	Dir. Acc.	Scen. Acc.	Avg. Ret.	Sharpe	Max DD
Zero-shot	51.2% ± 2.2	23.8% ± 0.8	0.16% ± 0.1	0.53 ± 0.38	34.5% ± 7.1
Reflective Accumulation	46.3% ± 2.9	23.3% ± 3.2	−0.08% ± 0.2	−0.12 ± 0.32	35.2% ± 11.4
Reflective Refinement	51.1% ± 1.8	24.7% ± 1.5	0.14% ± 0.2	0.36 ± 0.18	24.4% ± 9.0
Ours (full loop)	56.5% ± 1.0	29.0% ± 2.5	0.33% ± 0.1	1.00 ± 0.3	13.0% ± 4.4

Uncurated accumulation degrades 4.9pp below zero-shot with negative Sharpe.
Partial governance (Reflective Refinement) recovers accuracy to near zero-shot but Sharpe still lags (0.36 vs. 0.53).
Full loop: +5.3pp directional accuracy over zero-shot, Sharpe ~doubled vs. zero-shot, max drawdown reduced by ~62%.

Limitations¶

Validated on a single feedback domain (financial forecasting); generalization across heterogeneous feedback types (robotic control, binary task success, dense rewards) is unverified.
The critic, proposer, and curator are themselves LLM calls (Claude Sonnet 4.6), introducing inference cost per batch and potential for evaluation errors in the governance loop.
The paper is a workshop paper (ICML 2026 RLxF Workshop), so evaluation breadth is limited to 5 equities over one test year.
No ablation over batch size or curation frequency; the interaction between learning granularity and governance quality is unexplored.
The architecture's behavior when evidence logs grow very large (long-horizon deployment) is not characterized.

Relevance to Agentic AI / LLM Agents¶

This work directly addresses a critical failure mode of LLM agents that learn from environmental feedback without gradient updates—a paradigm increasingly used in tool-use, web browsing, and decision-making agents. The four-requirement framework (R1–R4) provides a concrete design checklist for any agent that accumulates context-as-memory across episodes, going beyond prior work that treats reflection as a one-time or monotonically growing operation. The separation of extraction (producing rules) from governance (managing rules' lifecycle and composition) is an architectural insight applicable to agentic systems well beyond financial forecasting—including long-horizon task agents, robotic agents, and multi-agent pipelines where rules or skills must be maintained across shifting environments. The finding that governance quality, not rule quantity, determines performance has direct implications for how memory and skill libraries in agentic frameworks (e.g., MemGPT-style systems) should be designed.