Skip to content

The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CGI (Critique-Guided Improvement) is a two-player actor-critic framework where a fine-tuned critic model generates structured natural language feedback to guide iterative refinement of an LLM agent's actions. A Llama-3-8B critic trained via distillation from GPT-4o surpasses GPT-4o itself as a critic by +29.16%, and the full CGI pipeline achieves state-of-the-art results across three interactive agentic environments.

Problem

Numerical reward signals (reward models, verifiers, Best-of-N) filter candidate actions but provide no contextual guidance for how to improve. Conversely, self-critique and self-refinement degrade in quality when the base model hallucinates or lacks diversity, and even well-trained LLM agents often fail to parse and act on verbal feedback correctly—a "poor utilization" problem that worsens after SFT.

Method

CGI operates as a two-stage, two-player framework over a POMDP:

  1. Critique Generation: A critic model πϕ (Llama-3-8B backbone) is fine-tuned via SFT on step-level expert critiques distilled from GPT-4o. Each critique has two structured components: Discrimination (evaluating candidate actions on Contribution, Feasibility, and Efficiency dimensions plus an overall grade) and Revision (actionable suggestions). Training data is collected only from correct trajectories (R(τ′)=1).

  2. Iterative Action Refinement: At each time step, the actor generates M=5 candidate actions; the critic evaluates each and returns critiques; the actor then produces a refined action conditioned on the critique. The actor is iteratively fine-tuned (K=3 iterations) on three data sources: Dcorrect (correct trajectories for reasoning), Drefine (critique-action pairs for critique utilization), and Dgeneral (ShareGPT for generalization). To avoid policy drift, the base model πθ (not the previous-iteration model) is fine-tuned each round.

Key Contributions

  • Two-player CGI framework separating actor (exploration) and critic (verbal feedback) with explicit training for both roles.
  • Structured critique format with three discrimination dimensions (Contribution, Feasibility, Efficiency) + graded revision.
  • Iterative action refinement SFT that resolves policy misalignment between training and inference.
  • Empirical demonstration that a small fine-tuned critic (8B) outperforms GPT-4o as a critic.
  • Analysis showing critiques are most impactful in early trajectory stages, and that CGI uniquely sustains gains on long-horizon tasks where iterative SFT plateaus.

Results

  • Critic quality (Table 1): CGI Critic-Llama-3-8B achieves average 61.44% vs. GPT-4o's 32.28% and DGAP's 23.64% as critic for Llama-3-8B actor (+29.16% over GPT-4o).
  • End-to-end CGI (Table 2): Llama-3-8B + CGI reaches 74.20% average (WebShop 76.17, ScienceWorld 78.43, TextCraft 68.00), vs. Iterative SFT 58.21%, GPT-4o 45.46%, AgentLM-70B 21.39%.
  • CGI outperforms GPT-4o by +28.74% average and AgentLM-70B by +52.81%.
  • Ablation: removing Drefine causes the largest drop (68.50% → 50.37%); ScienceWorld falls 26.94%.
  • Candidate action scaling: CGI improves from M=1 (56.89%) to M=5 (61.72%), saturating at M=7; self-critique shows no benefit from increasing M.

Limitations

  • Critic training requires GPT-4o as an expensive expert annotator and access to ground-truth expert trajectories per environment.
  • Experiments are limited to three text-based environments (WebShop, ScienceWorld, TextCraft); real-world GUI or open-ended settings are untested.
  • M=5 candidate actions per step multiplies inference cost fivefold at deployment time.
  • Fine-tuned actor models struggle to utilize critique even after SFT, requiring iterative refinement to recover—suggesting fundamental tension between instruction-following fine-tuning and receptiveness to external critique.
  • Evaluation on only 200/200/100 simulations per environment; no statistical significance reported.

Relevance to Agentic AI / LLM Agents

CGI directly addresses a core open problem in LLM agent training: how to generate and consume richer-than-scalar feedback during iterative self-improvement. The finding that SFT degrades critique uptake is a practically important warning for the field—standard fine-tuning pipelines may inadvertently reduce agent plasticity. The two-player separation of actor and critic echoes RLHF's reward model paradigm but replaces scalar scores with actionable natural language, which aligns better with how LLMs generate and process information. Results suggest critic specialization via distillation is a tractable path to surpassing frontier models (GPT-4o) at a fraction of the cost, with direct implications for agent training pipelines targeting long-horizon interactive tasks.