Joint Learning of Experiential Rules and Policies for Large Language Model Agents

🕒 Published (v1): 2026-06-25 15:11 UTC · Source: Arxiv · link

Why this paper was selected

Unified framework jointly learning in-context rules and parametric policies from agent experience

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

JERP jointly updates a natural-language experiential-rule pool and the LLM policy parameters from the same interaction trajectories, so the two stay mutually aligned throughout training. After each episode, sampled trajectories drive a GRPO-style parameter update and a contrastive-reflection rule-pool edit, letting stable behaviors gradually migrate into model weights while the rule pool adapts to the evolving policy. Experiments on AlfWorld and WebShop show consistent gains over both pure prompt-based and pure RL fine-tuning baselines.

Problem

Existing work treats experience reuse in one of two disjoint ways: (1) externalizing it as natural-language rules fed at inference time, or (2) burning it into model parameters via RL fine-tuning. The first suffers from rule staleness when the policy changes during training; the second provides sparse gradient signal for local, step-level mistakes. Neither approach coordinates rule maintenance with parameter learning, leaving rule pools misaligned with the policy they are meant to assist.

Method

JERP maintains a per-task long-term experiential-rule pool \(K(d) = \{(z_i, s_i)\}\) where each rule has textual content \(z_i\) and a utility score \(s_i\). The training loop has two coupled stages:

Stage I — Model-parameter update. For task instance \(d\), \(N\) complete trajectories \(\{\tau_i\}\) are sampled under the current policy \(\pi_{\theta_\text{old}}\) with the working rule set \(\tilde{K}(d) = \text{top-}k(K(d))\) prepended to the prompt. Group-relative advantages are computed as $\(A_i = \frac{R(\tau_i) - \mu_d}{\sigma_d + \delta}\)$ and parameters are updated with a clipped GRPO objective plus a KL penalty toward a reference policy.

Stage II — Experiential-rule update via contrastive reflection. A frozen LLM compares current trajectories \(T_d\) against accumulated reference successful trajectories \(T^+(d)\) and emits a structured edit list \(O(d)\) of five primitive operations: ADD, EDIT, UPVOTE, DOWNVOTE, MERGE. These are applied to \(K(d)\); utility scores are updated by fixed increments; rules falling below a score threshold are pruned; and the capacity limit is enforced via MERGE. The training objective couples both components: $\(J(\theta, K) = \mathbb{E}_{d \sim p(d)}\left[\mathbb{E}_{\tau \sim p_\theta(\tau | d, K)}[G(\tau)]\right].\)$

Key Contributions

• Unified training loop that updates both policy parameters and a natural-language rule pool from the same trajectory group, keeping them mutually aligned.
• Contrastive-reflection rule editor with five structured edit primitives (ADD / EDIT / UPVOTE / DOWNVOTE / MERGE), enabling pool maintenance (not just expansion) during online training.
• Score-based rule retention and pruning mechanism that gradually removes stale or low-value rules as the policy evolves.
• Demonstration that joint coupling improves over both prompt-only (Reflexion) and RL-only (GRPO/RLOO) baselines on AlfWorld and WebShop.

Results

• AlfWorld: JERP outperforms all five baselines (Vanilla LLM, ReAct, Reflexion, RLOO, GRPO) on task success rate across six task types (Pick, Clean, Heat, Cool, Look, Pick2); specific percentage numbers are cut off in the provided text excerpt.
• WebShop: JERP achieves higher average score and task success rate than baselines; exact figures are in the truncated table.
• Ablation confirms that continual rule-pool updating (as opposed to a static pool) contributes measurably to performance.

Limitations

• Rule-pool selection uses a simple top-\(k\) by utility score; no fine-grained instance-level relevance retrieval is performed, which the authors acknowledge may be suboptimal.
• Rule updates rely on a frozen LLM as the reflective editor; quality of edits is bounded by that model's ability to interpret contrastive trajectories.
• Reference successful trajectories \(T^+(d)\) are unavailable at the start of training, so early episodes receive no positive contrastive signal for rule updating.
• Evaluation is limited to two text-based environments (AlfWorld, WebShop); generalization to richer multimodal or tool-augmented settings is untested.
• The utility-score update scheme (fixed increments \(\delta_+\), \(\delta_-\)) is hand-tuned and may not transfer across domains.

Relevance to Agentic AI / LLM Agents

JERP directly addresses a core open problem in LLM agent training: how to efficiently accumulate and maintain experience across episodes without the rule bank and the policy drifting apart. By coupling GRPO-style RL fine-tuning with a dynamically maintained natural-language rule pool, it bridges the prompt-engineering and RL fine-tuning paradigms that the community has largely kept separate (cf. ExpeL, Reflexion, GiGPO). The contrastive-reflection rule editor is a practical mechanism for lifelong knowledge maintenance in agents that continuously learn, and the score-based pruning addresses rule staleness — a largely neglected issue. This work is directly relevant to anyone building agents that must learn across many task instances with sparse rewards, and the per-task rule pool design suggests a path toward modular, inspectable agent memory.