Skip to content

WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model

🕒 Published (v1): 2025-04-23 02:54 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WebEvolver augments iterative web-agent self-improvement with a co-evolving World Model LLM that (1) synthesizes multi-step training trajectories without real browser interactions and (2) performs look-ahead action selection at inference time. The world model is trained jointly with the agent policy on the same collected trajectories and grows alongside it. This yields a ~10% task-success gain over the prior OpenWebVoyager self-improving baseline across three live web benchmarks.

Problem

Self-improving web agents—which fine-tune on their own sampled trajectories via rejection sampling—plateau quickly. Two failure modes drive stagnation: the agent overfits to familiar trajectories (reduced exploration diversity), and inference-time tree-search remedies require expensive real browser rollouts for marginal gains. Prior world-model approaches (WebDreamer, WMA) use off-the-shelf frozen LLMs as world models purely at inference time and never leverage the world model to generate training data, limiting trajectory diversity and the depth of multi-step simulation.

Method

WebEvolver frames web navigation as a POMDP \((S, A, O, T, R)\) and co-trains two LLMs—an agent policy \(\pi_{\theta_M}\) and a world model \(\pi_{\theta_w}\)—on the same iteratively collected trajectories.

World Model Training. From each collected trajectory \(\tau = \{(o_0,a_0),\ldots,(o_T,a_T)\}\), a world-modeling trajectory \(\tau_w\) is derived where the target is to predict \(o_t\) given \(a_{t-1}\) and truncated history \(c^w_t\). A chain-of-thought rationale \(h^w_t\) is distilled from the base LLM to help generate coherent next-page accessibility trees. The objective is: $\(J(\theta_w) = \mathbb{E}_{\tau_w \sim D_i}\left[\sum_{t=1}^{T} \log \pi_{\theta_w}(o_t|c^w_t, h^w_t) + \log \pi_{\theta_w}(h^w_t|c^w_t)\right]\)$

Trajectory Synthesis. The world model replaces the real browser transition \(T\): synthetic observations \(\hat{o}_t \sim \pi_{\theta_w}(\cdot|I_w, c^w_t)\) are fed to the policy to generate synthetic actions \(\hat{a}_t\), producing synthetic trajectories \(\hat{\tau}\) of up to \(T=7\) steps. Only queries that failed real-environment self-improvement are used to generate synthetic data, maximizing diversity. Synthetic trajectories pass rejection sampling and are merged with real-environment SFT data to train the final WebEvolver policy.

World Model Look-Ahead (WMLA). At inference, for each candidate action \(a_t^{(k)}\) (up to 3 candidates sampled with temperature 0.7 plus prompt steering), \(d\)-step rollouts are generated via: $\(\hat{o}_{t+j} \sim \pi_{\theta_w}(\cdot|I_w,c^w_{t+j}), \quad (\hat{h}_{t+j},\hat{a}_{t+j}) \sim \pi_{\theta_M}(\cdot|I,q,\hat{c}^{\text{clip}}_{t+j})\)$ A GPT-4o scorer rates each rollout on a \(\{0, 0.5, 1.0\}\) scale; the highest-scoring action is executed.

Key Contributions

  • Co-evolving world model architecture: the world model is fine-tuned from the same open-source base (Llama-3.3-70B) and updated at each self-improvement iteration alongside the agent policy, rather than being a static off-the-shelf model.
  • Synthetic trajectory generation for training: world model substitutes for the real browser to produce multi-step (depth ≤ 7) training trajectories, scaling data without additional live-environment cost.
  • WMLA inference-time planning: multi-step look-ahead using co-evolved world model plus LLM scorer, achieving greater simulation depth than prior 1–3 step approaches.
  • Purely self-contained pipeline: no distillation from closed-source models; Llama-3.3-70B is used for both policy, world model, and rejection-sampling evaluator.

Results

  • WebVoyager (473 queries, text-only): WebEvolver reaches 42.49% success rate vs. 38.68% for self-improve iter 2 (the same-scale baseline); +WMLA (d=1) reaches 46.24%, vs. +WebDreamer at 44.61%.
  • Mind2Web-Live (53 queries): WebEvolver reaches 22.64% vs. ~18.86% for the base self-improving model; +WMLA (d=1) reaches 28.30%.
  • Both benchmarks show ~10% gain over OpenWebVoyager self-evolving baseline.
  • Intrinsic world model evaluation (Table 2, depth=1): structural correctness (STR) 41.24%, similarity 35.73%, overall 38.20% for base Llama; iter-2 world model reaches STR 75.96%, similarity 63.56%, overall 57.80% at depth=1—showing the co-evolved model substantially outperforms the frozen base on web-page prediction fidelity.
  • WMLA depth ablation: d=1 consistently best; deeper look-ahead (d≥2) degrades due to compounding world-model error.

Limitations

  • World model hallucinations accumulate at depth ≥ 2, causing WMLA to degrade with longer rollouts; the usable look-ahead horizon is practically limited to 1–2 steps.
  • WMLA uses GPT-4o as scorer at inference time, introducing a closed-source dependency and additional cost despite the training loop being fully open-source.
  • Evaluation on live websites is noisy: some sites were geographically filtered out, and results are averaged over two time windows to account for web-state churn.
  • Synthetic trajectory quality depends on the world model's fidelity; hallucinated web states may teach sub-optimal navigation habits even if rejection sampling filters complete failures.
  • Single benchmark domain (open web); transfer to structured environments (e.g., desktop GUIs, APIs) is untested.

Relevance to Harnesses / Meta-Harnesses

WebEvolver is a self-improving meta-harness in which the orchestration loop (collect → reject-sample → train → collect) is extended by a co-trained world model that acts as an internal synthetic environment—a "virtual web server" generating training episodes without any external tool call cost. This is directly analogous to meta-harness patterns where a harness manages multiple subsystems (here: policy LLM, world-model LLM, real browser, scorer) and routes work between them depending on the phase (training vs. inference). The WMLA mechanism—where the harness spins up simulated rollouts per candidate action before committing—is a concrete example of look-ahead planning integrated into the execution layer of an agent harness. For practitioners building multi-LLM harnesses, this work demonstrates that co-training a secondary model to simulate environment transitions can replace costly real-environment calls at both training and inference time, a reusable design pattern for any harness targeting high-cost external environments.