Skip to content

Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ExSearch is an agentic retrieval-augmented generation framework that trains LLMs to iteratively think, search, and record evidence using a generalized expectation-maximization (GEM) algorithm. The LLM self-improves by sampling search trajectories, weighting them by how well they support correct answers, and re-training on them—no external reward model required. On four multi-hop QA benchmarks it outperforms strong iterative RAG baselines by up to +7.8% exact match.

Problem

Complex, knowledge-intensive tasks (especially multi-hop QA) break standard RAG pipelines because: (1) single-shot complex queries yield low retrieval coverage, and (2) retrieved documents contain irrelevant content that misleads generation. Prior work cascades independent modules (query decomposition, re-ranking, extraction) trained with synthetic data in isolation, leaving end-to-end alignment unsolved and preventing a unified model from dynamically adapting retrieval as reasoning unfolds.

Method

ExSearch models agentic search as a latent-variable sequence: a trajectory z = {(x_i, d_i, e_i)} where the LLM alternates three actions—thinking (formulate a sub-query x_i), search (invoke external retriever, top-K docs d_i), and recording (extract fine-grained evidence e_i from retrieved docs). Training uses a GEM loop: - E-step (trajectory exploration): The current LLM samples multiple candidate trajectories via its own autoregressive generation; each trajectory z_i is assigned importance weight w(z) ∝ p(answer | x, z; θ_t) using Bayes' theorem—trajectories leading to correct answers get higher weight. - M-step (re-weighted trajectory learning): The LLM parameters are updated by maximizing the ELBO under importance-weighted trajectories, optimizing two objectives jointly: L_R (learn to reason/search, Eq. 7) and L_A (learn to aggregate and answer). Gradient update is standard SGD/DeepSpeed ZeRO 3. - Iterations alternate E/M until validation performance plateaus (early stopping). A cold-start with 1,000 pseudo-examples warms up the action-following capability before EM begins.

Key Contributions

  • ExSearch: a unified agentic framework that integrates dynamic retrieval, evidence extraction, and answer aggregation into a single LLM trained end-to-end.
  • GEM-based self-incentivization: importance-weighted trajectory learning without PPO/GRPO or an external verifier—cheaper than RL-based alternatives (e.g., Search-R1).
  • Convergence proof: Lemma 3.1 establishes non-decreasing log-likelihood per iteration; Theorem 3.2 (monotone convergence) guarantees a finite limit.
  • ExSearch-Zoo: extension suite covering LLaMA/Qwen families at 3B–24B scale, plus an extended action space adding list-wise document re-ranking (think→search→select→record).

Results

  • Main benchmark (Avg. F1 / EM / Acc.): Ours-Qwen-2.5-7B achieves 51.45 / 43.02 / 46.20 vs. Search-R1-Qwen-2.5-7B 48.74 / 39.89 / 44.02; vs. Search-o1-QwQ-32B 44.14 / 33.79 / 37.62.
  • HotpotQA F1: 62.59 (Ours-Qwen-2.5-7B) vs. 55.40 (RankRAG-LLaMA-3.1-70B) and 53.31 (Search-o1-QwQ-32B).
  • Peak EM improvement: +7.8% exact match over baselines (per abstract).
  • 7B vs. 70B: Outperforms GPT-4o and LLaMA-3.3-70B by +5.63 and +7.55 Avg. F1 despite 10× fewer parameters.
  • Ablation (HotpotQA F1, Qwen-2.5-7B): removing thinking: 54.23 (−8.36); removing search: 43.34 (−19.25); removing recording: 57.34 (−5.25); removing w(z): 57.23 (−5.36).
  • Recall@5 (HotpotQA): Ours-Qwen-2.5-7B 65.66 vs. ColBERTv2.0 standalone 51.79; vs. BGE re-ranking 55.89.
  • Scaling (ExSearch-Zoo): Mistral-2501-24B reaches Avg. F1 55.89 / EM 44.75 / Acc. 51.63; even 3B models (Qwen-2.5-3B) achieve competitive Avg. F1 40.85.
  • Extended re-ranking variant: +1.1% Acc. on MuSiQue, +1.5% on 2WikiQA over vanilla ExSearch.
  • Convergence confirmed empirically: peak performance typically at iteration 2–3 on held-out sets.

Limitations

  • Under-searching (3.5% of failure cases): model halts early on plausible but incomplete evidence, lacking trajectory-level stopping criteria.
  • Over-searching (7.5% of failure cases): model retrieves more docs than needed but still misses key facts; no adaptive depth control.
  • Cold start still required for best performance; zero warm-up lags K=1000 by a consistent margin (though still competitive).
  • Restricted to text-only, single-modality retrieval; authors flag multi-modal extension as future work.
  • Retrieval corpus fixed to 2018 Wikipedia dump; real-world deployment would need a live index.
  • Importance weights computed from string-match answer probability—can be noisy for long-form or non-factoid answers.

Relevance to Agentic AI / LLM Agents

ExSearch directly instantiates the core agentic loop—plan, act, observe, iterate—as a trainable system, replacing prompt engineering with principled EM-based self-improvement. It is a concrete example of how LLM agents can acquire tool-use skills (retrieval) without human-labeled trajectories or an external reward model, relevant to efforts on self-improving agents and autonomous search. The think/search/record action abstraction is easily extensible (re-ranking shown; future work: tool-augmented agents), making this a reusable framework pattern for broader agentic IR. The convergence guarantee distinguishes it from purely empirical RL training recipes (GRPO/PPO), offering a theoretically grounded alternative for iterative agent training.