Skip to content

ProRefine: Inference-Time Prompt Refinement with Textual Feedback

🕒 Published (v1): 2025-06-05 17:52 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ProRefine is an inference-time prompt optimization method that uses a three-LLM agentic loop—a task model, a feedback critic, and a prompt optimizer—to iteratively refine prompts for a specific query without any training data or ground-truth labels. Evaluated on five mathematical reasoning benchmarks, it surpasses zero-shot Chain-of-Thought baselines by 3–37 percentage points and outperforms the supervised TextGrad method in 11 of 15 dataset-model combinations.

Problem

Prior prompt optimization approaches either require offline fine-tuning with labeled data (e.g., TextGrad, DSPy) or rely on applying the largest available models universally, making them impractical for resource-constrained, dynamic deployments. No method existed for targeted, on-demand prompt repair at inference time when a standard prompt fails on a specific query.

Method

ProRefine runs a fixed agentic loop over three LLMs: \(\text{LLM}_\text{task}\), \(\text{LLM}_\text{feedback}\), and \(\text{LLM}_\text{optimizer}\). At each of up to \(n\) steps, \(\text{LLM}_\text{task}\) generates \(i \cdot k\) tokens of output for the current prompt \(p^*\) and query \(q\); \(\text{LLM}_\text{feedback}\) critiques the partial output and produces textual feedback \(f_i\); \(\text{LLM}_\text{optimizer}\) rewrites \(p^*\) incorporating \(f_i\). The loop terminates early on EOS detection. A verifier (here Llama3.1-70B-instruct serving dual roles via separate API calls) checks initial outputs and skips refinement when the answer is already correct. Hyperparameters \(k=10\) and \(n=25\) were fixed globally without per-task tuning. The method requires only API access, making it compatible with black-box LLMs.

Key Contributions

  • ProRefine: a training-free, label-free, inference-time prompt refinement algorithm using a three-LLM feedback loop.
  • Demonstration across five datasets (BIG-Bench Hard object counting, word sorting, GSM8K, SVAMP, AQUARAT) with three task models (Llama3.2-1B/3B, Llama3.1-8B).
  • Empirical analysis of verifier impact: comparing no-verifier, LLM-verifier, and optimal (oracle) verifier conditions to bound the loop's potential.
  • Evidence that smaller task models can approach zero-shot performance of models one tier larger via ProRefine.

Results

  • Object Counting: +3–16 pp over CoT; beats TextGrad on 2/3 models (up to +3 pp); Llama3.1-8B with verifier: 0.89 vs. CoT 0.73.
  • Word Sorting: +8–37 pp over CoT; beats TextGrad on 1/3 models; Llama3.1-8B with verifier: 0.71 vs. CoT 0.50.
  • GSM8K: +2.4–20.4 pp over CoT; beats TextGrad on all 3 models by 2.1–19.1 pp; Llama3.2-1B with verifier: 0.654 vs. CoT 0.450.
  • SVAMP: +0.9–11.9 pp over CoT; beats TextGrad across all models; Llama3.2-1B with verifier: 0.808 vs. CoT 0.689.
  • AQUARAT: Mixed; +0.9–2 pp over CoT on some models, but −5.4 to −22 pp on others; beats TextGrad on 2/3 models.
  • Optimal verifier achieves best results in 13/15 cases, revealing verifier quality as the dominant bottleneck.
  • ProRefine outperforms TextGrad overall in 11/15 cases, despite TextGrad using full train+validation sets.

Limitations

  • Performance is inconsistent: notable degradation on AQUARAT and Word Sorting for smaller models (Llama3.2-1B/3B).
  • Verifier quality is a hard ceiling; the gap between LLM-verifier and optimal-verifier is substantial, and no bespoke verifier is trained.
  • Hyperparameters \(k\) and \(n\) are fixed globally without per-task tuning, leaving task-specific gains on the table.
  • Relies on a large capable model (Llama3.1-70B) for feedback, optimization, and verification, which imposes latency and cost overhead for every refined query.
  • Evaluated only on mathematical/symbolic reasoning; generalization to open-ended generation tasks is undemonstrated.
  • TextGrad baseline uses supervised data, making the comparison asymmetric—ProRefine's advantage is partly explained by the supervised training regime, not just architectural differences.

Relevance to Harnesses / Meta-Harnesses

ProRefine is a concrete example of a meta-harness pattern: a fixed orchestration loop that wraps an arbitrary task-executing LLM with critic and optimizer agents to improve its instructions dynamically, without any external scaffolding framework. The three-LLM architecture (\(\text{LLM}_\text{task}\), \(\text{LLM}_\text{feedback}\), \(\text{LLM}_\text{optimizer}\)) directly instantiates a harness over a base agent, where the harness itself generates and applies corrective signals to the agent's prompt—analogous to how a meta-harness manages and tunes sub-agent behavior. The work empirically quantifies how much performance gain is attributable to loop quality (verifier accuracy) versus loop structure, which is precisely the diagnostic question relevant to harness designers optimizing agent orchestration pipelines.