Skip to content

LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Razvan Pascanu (DeepMind); RL fine-tuning makes LLM agents greedy, undermining decision-making

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLMs acting as decision-making agents exhibit three systematic failure modes — greediness, frequency bias, and the knowing-doing gap — that prevent them from exploring effectively in bandit and game-playing settings. The authors apply Reinforcement Learning Fine-Tuning (RLFT) on self-generated Chain-of-Thought rationales to mitigate these failures, demonstrating measurable improvements in exploration and regret across Gemma2 model families.

Problem

LLM agents underperform in sequential decision-making despite possessing relevant world knowledge: they prematurely commit to greedy exploitation (up to 55% of the action space left unexplored), small models copy the most frequent action in context regardless of reward (frequency bias), and large models correctly compute optimal strategies in CoT yet still execute suboptimal greedy actions — the knowing-doing gap. Prior fine-tuning work relied on expert trajectories (imitation learning); the gap between in-context reasoning and action execution was unquantified.

Method

RLFT pipeline: A pre-trained LLM \(\pi_\theta\) interacts with the environment, generating action tokens \(z_t = [z_t^{\text{CoT}}; a_t]\) — a CoT reasoning trace followed by an extracted action. The context at step \(t\) is \(c_t = [c_t^{\text{in}}; c_t^{\text{out}}; \tau_{t-C:t}]\), where \(\tau\) is the last \(C\) (state, action, reward) triples. The fine-tuning objective is PPO with clipping and a KL penalty to the reference policy:

\[\max_\theta \mathbb{E}_{(c,z)\sim\mathcal{D}} \left[\min\!\left(\frac{\pi_\theta(z|c)}{\pi_{\theta_{\text{old}}}(z|c)}A^{\text{adv}},\, \text{clip}_\epsilon\!\left(\frac{\pi_\theta(z|c)}{\pi_{\theta_{\text{old}}}(z|c)}\right)\!A^{\text{adv}}\right) - \beta D_{\text{KL}}(\pi_\theta(\cdot|c)\|\pi_{\text{ref}}(\cdot|c))\right]\]

Advantages use Monte Carlo returns-to-go for fixed-horizon bandits and GAE with a learned value head for variable-length Tic-tac-toe. Invalid actions incur a shaped penalty \(r_t^{\text{valid}} = -5\). Models trained: Gemma2 2B and 9B for 30K gradient steps on pools of 512 randomly drawn MAB instances. Exploration augmentations studied: \(\epsilon\)-greedy action injection, self-consistency (majority vote over \(N\) rollouts), and self-correction (re-prompting on previous failures). Expert CoT data ablations also explored.

Key Contributions

  • Systematic quantification of three LLM decision-making failure modes: greediness (up to 55% action space unexplored), frequency bias (96% frequent-action fraction in Gemma2 2B), and the knowing-doing gap (87% correct rationales but only 21% optimal actions in Gemma2 27B).
  • RLFT on self-generated CoT rationales as a mitigation strategy, showing it increases action coverage (+12% for 2B) and lowers cumulative regret across 5/10/20-arm Gaussian and Bernoulli bandits and CBs.
  • Empirical evaluation of classic (\(\epsilon\)-greedy) and LLM-specific (self-consistency, self-correction) exploration augmentations for more effective RLFT.
  • Ablations isolating the value of CoT reasoning, expert data bootstrapping, and extended reasoning token budgets (\(G\)) on decision-making performance.

Results

  • Greediness: Gemma2 27B (without RLFT) covers only 65% of arms in a 10-arm MAB and 45% in a 20-arm MAB; RLFT on 2B increases coverage by +12% after 30K updates.
  • Frequency bias: Gemma2 2B has 96% frequent-action fraction; RLFT reduces it from 70% → 35% at low repetitions (0–10), though it remains elevated at high repetitions.
  • Knowing-doing gap (Gemma2 27B): 87% correct UCB rationales, yet 58% of actions on correct rationales are greedy; only 21% are truly optimal.
  • Regret (medium noise \(\sigma=1\), button MAB): RLFT lowers cumulative regret for both 2B and 9B across 5/10/20 arms relative to ICL baselines; UCB remains the upper-bound reference.
  • Results replicated on Llama3 and Qwen2.5 families (Appendix C.4), confirming failure modes generalize across architectures.
  • Tic-tac-toe: RLFT improves win rate over a random opponent, consistent with bandit findings (frontier models without RLFT barely beat random per Ruoss et al. 2024).

Limitations

  • Evaluated only on simple/toy environments (multi-armed bandits, contextual bandits, Tic-tac-toe); generalization to complex real-world agentic tasks is unverified.
  • RLFT does not fully eliminate frequency bias (remains elevated at high context repetitions) or close the knowing-doing gap entirely.
  • Exploration after RLFT remains sub-optimal relative to UCB; RLFT is not claimed as a complete solution.
  • Models studied are small-to-medium scale (2B–27B); behavior of frontier-scale models under RLFT is not directly investigated.
  • The analysis of the knowing-doing gap is specific to the UCB algorithm task construction and may not transfer cleanly to other domains.

Relevance to Agentic AI / LLM Agents

This paper directly targets the foundational question of why LLM agents fail at sequential decision-making, providing concrete diagnostic metrics (action coverage, frequency fractions, knowing-doing confusion matrices) that are reusable across agent evaluations. The knowing-doing gap — a model correctly reasoning yet acting suboptimally — is a critical failure mode for any agentic system that relies on CoT for planning. The RLFT approach connects to the broader RLHF/GRPO line of work (DeepSeek-R1, Zhai et al.) but applies it specifically to exploration in interactive settings rather than reasoning benchmarks, offering a diagnostic-first perspective that complements purely performance-driven fine-tuning work.