When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Emergent exploitation bias in LLM agents trained via RL on bandit tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Training LLMs via SFT or RL on multi-armed bandit (MAB) tasks produces meta-bandit agents that match UCB/Thompson Sampling in average regret and generalize to 6× longer horizons, but behavioral analysis reveals the gains come from more sophisticated exploitation rather than better exploration. RL-trained agents and SFT agents both exhibit higher suffix failure rates than pretrained models, indicating premature abandonment of exploration. The work advocates moving beyond average regret as the sole evaluation metric for LLM exploration policies.

Problem

LLMs default to greedy, short-sighted exploitation in sequential decision-making, performing poorly on multi-armed bandit tasks. While prior work applies SFT or RL to improve performance, it is unclear how these paradigms mechanistically shape exploration strategies, whether improvements reflect genuine exploration learning or just better exploitation, and how well the resulting policies generalize out-of-distribution and to longer horizons.

Method

The paper trains Qwen 2.5 (3B and 7B) on K-armed MAB tasks using two paradigms:

RL (PPO): Frames the MAB as a two-level hierarchical MDP with a high-level turn policy and low-level token policy. Token-level advantages are computed via a dual-scale GAE with separate discount/trace-decay coefficients \((\gamma_{\text{intra}}, \lambda_{\text{intra}})\) and \((\gamma_{\text{inter}}, \lambda_{\text{inter}})\) for intra-turn and inter-turn dynamics. Three reward signals are studied: - RL-OG: Original stochastic bandit reward. - RL-STG (strategic): Regret-normalized reward \(\tilde{r}_t = \frac{\mu_{A_t} - \min_i \mu_i}{\mu^* - \min_i \mu_i} \in [0,1]\), reducing variance via pseudo-regret baseline subtraction. - RL-ALG (algorithmic): Binary reward \(r_t = \mathbf{1}[\text{agent action} = \pi_{\text{UCB}}(o_t)]\), enabling RL-based oracle imitation without needing to learn a reward model.

SFT: Full fine-tuning on 32k UCB-generated trajectories with explicit CoT demonstrations of UCB value calculations (step-by-step \(\sqrt{\ln(t)/N_t(a)}\) computations).

Observation is presented as per-arm summary statistics (pull count, average reward) rather than raw history. Evaluation uses cumulative regret at \(t \in \{50, 300\}\), time-averaged reward, best-arm frequency, suffix failure rate, and greedy frequency, across Gaussian and Bernoulli bandit families.

Key Contributions

• Unified comparative study of SFT vs. RL for training LLMs as meta-bandit agents, with systematic OOD and length-generalization evaluation.
• Two principled reward designs: strategic (regret-shaped) reward that reduces training variance in high-variance Gaussian environments, and algorithmic (UCB-imitation) reward that enables efficient oracle imitation via binary signal.
• Finding that RL-ALG agents trained to imitate UCB consistently outperform their UCB teacher, by learning more exploitative UCB variants that prune poorly-performing arms earlier.
• Behavioral analysis using suffix failure rate revealing that all fine-tuned agents develop exploitation bias and are more prone to catastrophic early failure than pretrained models.
• Demonstration that RL policies generalize more robustly across bandit families (cross-distribution transfer) than SFT, while SFT shows fragility due to arithmetic overfitting to training distributions.

Results

• In-distribution: All fine-tuned 7B models achieve cumulative regret comparable to UCB (\(C=0.5\)) and Thompson Sampling at \(t=50\) and \(t=300\) on both Gaussian and Bernoulli training environments; pretrained model exhibits linear regret similar to Greedy.
• Length generalization (6×): RL and SFT agents trained on \(T=50\) maintain competitive regret at \(T=300\).
• OOD transfer: RL-ALG is the most robust across distribution families; SFT trained on Bernoulli5 shows degraded worst-case performance on Gaussian environments.
• Exploitation bias: All fine-tuned agents show non-zero suffix failure rates (Table 2); RL-ALG has suffix failure rate ~10.5% vs. ~0% for UCB at \(t=50\) in-distribution; greedy frequency rises significantly post fine-tuning.
• RL-ALG vs. UCB teacher: RL-ALG matches or achieves lower cumulative regret than the UCB teacher across all evaluated environments.
• Strategic reward (RL-STG): Improves over RL-OG in Gaussian settings (higher variance); performance gap narrows in Bernoulli settings where variance is lower.

Limitations

• Experiments are limited to Qwen 2.5 3B and 7B; results may not generalize to larger-scale or architecturally different LLMs.
• The MAB testbed, while canonical, is a simplified abstraction; whether these exploitation-bias findings transfer to richer agentic tasks (tool use, web navigation) is untested.
• Suffix failure rate is a suggestive surrogate metric, not a direct measure of exploration quality; formal worst-case regret bounds are not derived.
• RL-ALG requires access to an oracle algorithm (UCB) during training, which may not be available in general sequential decision-making settings.
• RL-OG and RL-STG show instability in OOD evaluation; root causes are not fully characterized.
• Evaluation uses only 64 episodes per policy due to LLM inference cost; this limits statistical power relative to classical bandit experiments with thousands of rollouts.

Relevance to Agentic AI / LLM Agents

This paper directly investigates a core capability requirement for LLM agents: principled exploration vs. exploitation in sequential decision-making, using MAB as a controlled proxy. The finding that fine-tuning—whether via SFT or RL—induces exploitation bias rather than robust exploration is a significant warning for agentic system design: standard training objectives optimize average returns but systematically weaken the worst-case exploratory robustness that long-horizon autonomous agents require. The dual-scale PPO formulation for token-level credit assignment in multi-turn LLM interactions is broadly applicable beyond bandits to any agentic RL loop. The result that RL-ALG agents surpass their UCB teacher by learning more aggressive exploitation variants has direct implications for using LLMs as meta-learners in tool-selection, API-calling, and planning agents, where over-exploitation of apparently good actions is a known failure mode.