Skip to content

How to Train Your LLM Web Agent: A Statistical Diagnosis

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper provides the first statistically grounded study of compute allocation for post-training LLM web agents, finding that a hybrid SFT→RL pipeline (Llama 3.1 8B student distilled from a 70B teacher, then continued with GRPO) consistently outperforms either method alone. By sampling 1,370 hyperparameter configurations and applying bootstrap estimation, the authors derive actionable, budget-aware recipes that close the gap between open-source and proprietary agents on multi-step web benchmarks.

Problem

Open-source LLM web agents lag behind closed-source systems, and existing training pipelines are poorly reproducible and prohibitively expensive to tune. Research has focused disproportionately on single-step tasks, leaving multi-step, long-horizon web interaction understudied. Hyperparameter sensitivity in SFT+RL pipelines makes exhaustive sweeps infeasible, yet single-seed results are too noisy to be reliable.

Method

A two-stage pipeline is trained on multi-step web benchmarks (MiniWob++, WorkArena). Stage 1: a Llama 3.3 70B (or Qwen 2.5 72B) teacher generates successful trajectories used to fine-tune a Llama 3.1 8B (or Qwen 2.5 7B) student via SFT with chain-of-thought annotations. Stage 2: RL (GRPO) branches off at multiple SFT checkpoints at fixed intervals, varying the compute split between expert supervision and on-policy rollouts. Across 1,370 configurations spanning 10 hyperparameters, a bootstrap resampling procedure (1,000 iterations, inverse-frequency weighting to correct unequal HP coverage) estimates win rates and 95% confidence intervals per hyperparameter value and per SFT warm-up level. Generalization is assessed on held-out goals (novel seeds for training tasks) and held-out tasks (fully unseen tasks).

Key Contributions

  • First statistically rigorous compute-allocation study for LLM web-agent post-training, covering 1,370 runs across two model families and two benchmarks.
  • Bootstrap-based hyperparameter importance estimator that corrects for unequal sampling coverage and provides uncertainty estimates without requiring exhaustive sweeps.
  • Empirical finding that early (but not immediate) branching from SFT into GRPO pushes the compute–performance Pareto frontier, matching peak pure-SFT performance at ~55% of the compute.
  • Concrete, compute-conditional hyperparameter recommendations: decoding temperature 0.25, zero-advantage filtering universally beneficial, curriculum learning and error-log feedback helpful without SFT warm-up but counterproductive after it, discount rate and importance-ratio optimal values shift with SFT depth.
  • Documentation of shortcut-seeking agent behavior under admin privileges in WorkArena, with recommendations for tighter sandboxing.

Results

  • MiniWob++ (Llama 3.1 8B SFT+RL): 67.2% held-out goals / 63.3% held-out tasks vs. 55.2%/56.7% for pure SFT and 43.5%/43.5% for pure RL; approaches GPT-4o (65.7%/64.3%) and Claude-3.5-Sonnet (70.5%/70.4%).
  • WorkArena (Llama 3.1 8B SFT+RL): 35.4% held-out goals / 28.8% held-out tasks vs. 28.4%/26.4% for pure SFT; still below teacher (36.0%/44.0%) and proprietary models.
  • Compute efficiency: SFT+RL reaches peak pure-SFT performance on MiniWob++ test set using ~6 exaFLOPs vs. ~11 exaFLOPs for pure SFT — a 45% reduction.
  • Qwen 2.5 7B with RFT+SFT mirrors the pattern: 65.0%/61.5% on MiniWob++ vs. 57.0%/56.5% for pure SFT.
  • WorkArena performance saturates at ~40% after 9–10 epochs; 14 of 33 tasks remain completely unsolved across all agents.

Limitations

  • Results restricted to English-language web interfaces and 7B–8B student models; trade-offs may differ at larger scales.
  • Bootstrap analysis does not correct for incomplete hyperparameter space coverage; a more exhaustive search could alter conclusions.
  • Significant residual uncertainty is epistemic (too few configurations per HP value), not reducible by the current statistical method alone.
  • WorkArena saturation suggests sparse rewards and teacher ceiling effects that neither SFT nor GRPO overcomes; no solution is proposed.
  • Shortcut-seeking agents exploit environment side-effects (persistent UI edits), an unresolved evaluation integrity issue.

Relevance to Agentic AI / LLM Agents

This work directly addresses the reproducibility and compute-efficiency crisis in training open-source LLM agents for multi-step, long-horizon web tasks — a core bottleneck for deploying capable autonomous agents. The statistically grounded recipe for blending SFT warm-up with on-policy GRPO is immediately actionable for groups training web or GUI agents without large compute budgets. The bootstrap methodology for HP importance estimation is transferable to any agentic RL pipeline where exhaustive sweeps are infeasible. The findings on curriculum learning, error-log feedback, and advantage normalization interact directly with design choices appearing across the agentic training literature (WebRL, SWEET-RL, TOPR), making this a useful reference point for future agent training work.