Skip to content

Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RL

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PNLC (Planning with a Natural Language Critic) trains a lightweight, goal-conditioned value function offline using IQL, then uses it at inference to generate natural-language critiques of an LLM agent's proposed reasoning steps. The agent iteratively refines its high-level thought strategies based on predicted positive and negative outcomes—without any inference-time search or direct LLM fine-tuning.

Problem

Multi-turn RL fine-tuning of frontier LLMs (GPT-4, Claude) is computationally prohibitive and requires API access to model weights. Inference-time search methods (MCTS, Reflexion, LATS) improve planning but incur 10× latency overhead. Neither approach scales effectively: RL fine-tuning is restricted to small models (GPT-2), while search is too slow for time-sensitive tasks.

Method

Offline training: Given a dataset of suboptimal trajectories, PNLC extracts (state, thought, next-state, goal) tuples, summarizes them, embeds them with GPT-3, and trains a goal-conditioned Q-function using a modified IQL objective: Q̂(s, aᵗʰᵗ, g) predicts the probability of reaching goal state g from state s after thought aᵗʰᵗ. The Q-function is a 2-hidden-layer MLP over embeddings—not a Transformer.

Inference (natural language critic): At each step, an LLM generates n=4 hypothetical future goals (2 positive, 2 negative). The MLP Q-function scores each. Goals and their likelihoods are formatted into a natural language value summary and fed back to the GPT-4 agent, which refines its thought. Two refinement rounds (m=2) are used in practice. No rollouts or environment interaction occur during inference.

The key abstraction: thoughts (high-level strategies, e.g., "appeal to credibility") are evaluated rather than raw utterances, reducing decision space complexity.

Key Contributions

  • Goal-conditioned natural language value function trained via offline IQL that outputs interpretable outcome probabilities rather than scalar rewards.
  • Operating over high-level thought abstractions (chain-of-thought reasoning steps) rather than low-level environment actions, enabling a lightweight MLP critic instead of a full LLM.
  • Inference procedure that evaluates both positive and negative hypothetical futures to drive self-refinement, replacing search with data-driven critique.
  • Demonstrated scalability to closed API-only frontier models (GPT-4) with no weight access required.

Results

All results over 100 test runs:

  • WebShop: Score 78.2 / SR 48% — beats Agent Q n=30 (77.1/44%), LATS n=30 (74.9/44%), ArCHeR (62.3/32%), ReAct (55.1/27%). Inference: 5s vs. 46s for Agent Q n=30.
  • AvalonBench (Merlin win rate): 47% — beats Strategist n=30 (42%), LATS n=30 (38%), Reflexion (26%), ReAct (21%). Inference: 6s vs. 62s for Strategist n=30.
  • Persuasion (avg. donation): \(0.87 — beats GDP-Zero n=30 (\)0.78), Reflexion (\(0.54), ReAct (\)0.54). Baseline dataset avg: $0.21.
  • Ablations: Removing goal-conditioning (scalar Q) drops to ReAct-level performance (78.2→55.4 score, 47→25% win rate). Removing refinement in favor of best-of-5 selection also collapses to near-ReAct performance. Both components are necessary.
  • Uses <10% the inference budget of the next-best task-specific methods.

Limitations

  • Value functions are task-specific and must be retrained per domain; no generalist critic is demonstrated.
  • Relies on LLM intuition to hypothesize plausible future states; may fail on domains far outside LLM pretraining (e.g., medical diagnosis requiring deep domain knowledge).
  • Quality and diversity requirements of offline data are not systematically characterized.
  • Experimental comparisons involve heterogeneous base LLMs across methods (GPT-2 for ArCHeR, Mixtral-7B for Agent Q, GPT-4 for PNLC/reasoning baselines), making iso-model ablations impractical.

Relevance to Agentic AI / LLM Agents

PNLC offers a scalable alternative to the two dominant paradigms for improving agent planning—RL fine-tuning and inference-time search—by decoupling value learning from the agent policy. This is directly relevant to the growing line of work on making LLM agents more capable in multi-turn interactive settings (negotiation, tool use, social reasoning) where single-step reasoning fails. The offline-trained natural language critic pattern generalizes across task types with only a lightweight auxiliary module, making it practical for closed-API frontier models—a key constraint in real-world agent deployments. The ablation results sharply identify that both goal-conditioned language descriptions and iterative refinement are necessary, offering concrete design guidance for future agent architectures.