Skip to content

Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM agents for sequential decision-making can autonomously improve by accumulating their own successful trajectories as in-context examples, without any task-specific prompt engineering. A naive bootstrapping approach already yields large gains (e.g., ALFWorld 73%→89%); adding database-level and exemplar-level curation pushes this to 93%, surpassing systems that use stronger LLMs and hand-crafted components.

Problem

Improving LLM agents for sequential decision-making typically requires labor-intensive, task-specific knowledge engineering: curated human examples, custom prompts, and bespoke observation/action spaces. This approach scales human effort rather than agent capability, and provides no mechanism for autonomous self-improvement from experience.

Method

The paper proposes Traj-Bootstrap and two curation enhancements on top of a ReAct-style agent with dynamic per-decision-point retrieval (plan/reason/act loop, Algorithm 1):

  • Traj-Bootstrap: Starts from a minimal set of human examples; after each training task, successful trajectories (reward = 1) are appended to a retrieval database D. Future tasks retrieve the k most relevant trajectory segments at each decision point via embedding similarity.

  • +DB-Curation (Algorithm 2): Maintains N parallel database instances. At exponentially spaced checkpoints (10, 20, 40, … tasks), the worst-performing database (by rolling success rate) is replaced with a copy of the best. This propagates emergent database-level properties (coverage, diversity, complementarity) via population-based training.

  • +Exemplar-Curation (Algorithm 3): Pools all trajectories across N databases and scores each by a retrieval-weighted quality metric Q(Ï„) = Σ(oᵢ·fáµ¢(Ï„)) / Σfáµ¢(Ï„), where oáµ¢ is binary task outcome and fáµ¢(Ï„) is retrieval frequency during task i. Constructs a composite database by selecting the top-1 trajectory per training task by Q. This is analogous to value-function estimation in RL.

The two curations are complementary and can be combined (+DB+Exemplar-Curation). All methods use identical test-time compute; only training differs.

Key Contributions

  • Demonstrates that naive trajectory bootstrapping (no curation) substantially outperforms fixed human-curated baselines across three benchmarks.
  • Introduces a population-based database-level curation (+DB-Curation) that selects and propagates high-performing trajectory collections.
  • Introduces a retrieval-weighted value metric for exemplar-level curation (+Exemplar-Curation) that identifies individual trajectories by their empirical utility as in-context examples.
  • Shows that the performance gain from bootstrapping exceeds the gain from upgrading GPT-4o-mini to GPT-4o on ALFWorld.
  • Demonstrates cross-model transfer: databases collected with GPT-4o-mini improve Mixtral 8x7B by up to +28 points.
  • Shows collected trajectories can also serve as fine-tuning data (ReAct-Finetune outperforms in-context approach on 2/3 benchmarks).
  • Provides cost analysis showing break-even vs. GPT-4o at 1,100 tasks; million-task/day savings of ~$530K vs. GPT-4o.

Results

  • ALFWorld: Fixed-DB 0.73 → Traj-Bootstrap 0.89 → +DB+Exemplar-Curation 0.93 (gpt-4o-mini); surpasses AutoManual (0.91, uses gpt-4-turbo+gpt-4o-mini with hand-crafted obs/action spaces) and AutoGuide (0.79).
  • InterCode-SQL: Fixed-DB 0.75 → Traj-Bootstrap 0.79 → +DB+Exemplar-Curation 0.82.
  • Wordcraft: Fixed-DB 0.55 → Traj-Bootstrap 0.64 → +Exemplar-Curation 0.72 (best single method); +DB+Exemplar-Curation 0.69.
  • Traj-Bootstrap (single attempt) matches Fixed-DB pass@2–pass@3 on all benchmarks; +DB+Exemplar-Curation approaches Fixed-DB pass@4–pass@5.
  • Cross-model transfer (GPT-4o-mini DB → Mixtral 8x7B): +28 pts ALFWorld, +18 pts InterCode-SQL, +12 pts Wordcraft.
  • Fine-tuning on self-collected trajectories: +23 pts ALFWorld (vs. +20 in-context), +19 pts Wordcraft (vs. +14), but −3 pts on InterCode-SQL.
  • Success predictor (Random Forest on embeddings): AUROC 0.77 (InterCode-SQL), 0.71 (Wordcraft).

Limitations

  • Only successful trajectories are stored; failed trajectories are discarded due to the credit attribution problem, leaving potentially useful negative signal unused.
  • +DB-Curation shows performance dips at small database sizes due to noisy quality estimates from low sample counts.
  • Diminishing returns set in after the first ~25% of training tasks; per-example utility declines as database grows.
  • +DB-Curation does not consistently help on Wordcraft; combining it with +Exemplar-Curation on Wordcraft underperforms +Exemplar-Curation alone (0.69 vs. 0.72).
  • Training cost for the full configuration is ~$600 (5 databases × 3,500 tasks on GPT-4o-mini); requires non-trivial upfront investment before break-even.
  • Evaluation confined to three benchmarks with large task pools; applicability to low-task-count domains is unclear.
  • Assumes access to binary success signal at training time; not applicable when environment reward is unavailable.

Relevance to Agentic AI / LLM Agents

This paper directly addresses the core bottleneck in deploying LLM agents: the dependence on human-curated, task-specific knowledge. By framing trajectory curation as an analogue of RL value estimation and population-based training, it offers a principled, model-agnostic self-improvement loop that scales with compute rather than human effort. The result that naive accumulation alone rivals a model upgrade is a practically significant finding for anyone designing agentic pipelines, as it suggests that the retrieval database is an underappreciated axis of optimization. The cross-model transferability of curated databases also points toward a separation of concerns—collect trajectories with a capable model, deploy inference cheaply—that is relevant to production agentic system design.