Skip to content

AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentTrek is a fully automated three-stage pipeline that harvests web tutorials, converts them into structured task specifications, and executes them with a VLM agent in real browser environments to synthesize GUI agent trajectory data at scale. It produces 10,398 multimodal trajectories at $0.55 each without human annotation. Models fine-tuned on AgentTrek data achieve state-of-the-art results on WebArena, ScreenSpot, and Multimodal-Mind2Web.

Problem

High-quality GUI agent trajectory data—comprising interleaved observations, natural language reasoning, and grounded actions—is scarce and expensive to collect via human annotation. Existing datasets top out at ~2,500 trajectories; human annotation is unscalable and does not generalize across the breadth of real web environments.

Method

The pipeline has three stages:

  1. Tutorial collection: Starting from the 20.8B-entry RedPajama corpus, a rule-based pre-filter (keyword, length, URL heuristics) reduces it to 68.8M entries; GPT-4o-mini labels 90k samples (F1=88.5%); a FastText classifier (F1=89.5%) trims the set to ~18.8M; GPT-4o-mini then standardizes each tutorial into platform, task description, prerequisites, step-by-step instructions, and expected outcome fields.

  2. Guided replay: A VLM agent (GPT-4o) executes each structured tutorial inside BrowserGym/Chromium, observing viewport screenshots and the AXTree (not raw HTML). All screenshots, AXTree snapshots, Playwright action traces, and chain-of-thought reasoning are recorded. A separate VLM evaluator (GPT-4o, 84.0% accuracy vs. human-annotated gold set) filters trajectories by instruction adherence and task completion.

  3. Model training: Vision-based agents use Qwen2-VL-7B fine-tuned on 10k trajectories with pyautogui pixel-level actions. Text-based agents use Qwen2.5-7B/32B fine-tuned on 6k trajectories with Playwright AXTree-based actions.

Key Contributions

  • End-to-end automated pipeline converting internet tutorials into verified GUI trajectories—no human annotators required.
  • Dataset of 10,398 trajectories spanning 127 websites and 11 task categories, with full multimodal coverage (screenshots, video, AXTree, DOM, reasoning chains, avg. 12.1 steps/trajectory).
  • Cost of $0.55/trajectory vs. expensive human labeling.
  • Demonstrated that guided replay (52% task success) outperforms prompting with high-level goals alone (15.78% success) for trajectory quality.
  • Scaling analysis showing synthetic data approaches human-annotated Mind2Web performance as volume increases.

Results

  • WebArena (task success rate): Qwen2.5-7B + AgentTrek → 10.46%; Qwen2.5-32B + AgentTrek → 22.40%, surpassing GPT-4o (13.10%) and GPT-4 (14.41%) baselines.
  • ScreenSpot Web (visual grounding, average): Qwen2-VL-7B baseline 30.7% → 67.4% with AgentTrek (+36.7 pp), surpassing CogAgent (50.7%) and approaching GPT-4+OmniParser (67.0%).
  • Multimodal-Mind2Web cross-domain step SR: Qwen2-VL baseline 42.1% → 47.7% with AgentTrek → 52.6% with AgentTrek+Mind2Web.
  • Scaling from 20% to 100% of AgentTrek data improves cross-domain step SR from 39.5% to 45.0%; full human-annotated Mind2Web achieves 47.7% on this metric.

Limitations

  • Pipeline relies on live websites; dynamic GUI changes or site unavailability can break replay and reduce trajectory yield.
  • Task coverage limited to 11 categories and 127 websites; broader or domain-specific coverage requires re-running the pipeline.
  • Guided replay still uses GPT-4o (costly proprietary model) as the executor; $215 per 1,000 tasks limits very large-scale runs.
  • VLM evaluator applies more stringent criteria than humans, potentially over-filtering valid trajectories.
  • No evaluation on desktop or mobile GUI tasks despite the multimodal data schema supporting them.

Relevance to Vision-Language Models

AgentTrek is a direct demonstration of VLMs operating as both executor and evaluator in an agentic data-generation loop—showing that modern VLMs (GPT-4o, Qwen2-VL) are capable enough to close the loop on trajectory synthesis without human oversight. The pipeline directly benefits VLM-based GUI agents by providing rich, grounded, multimodal training signal (screenshots + AXTree + reasoning chains) that standard pretraining corpora lack. For researchers tracking VLMs, the key takeaway is that guided replay with structured language artifacts (tutorials) dramatically improves VLM task-execution success rates compared to open-ended prompting, reinforcing the importance of structured grounding for agentic VLM applications. The work also validates Qwen2-VL as a competitive vision backbone for GUI grounding when fine-tuned on domain-matched trajectory data.