Skip to content

Evaluating Long-Context Reasoning in LLM-Based WebAgents

🕒 Published (v1): 2025-12-03 22:53 UTC · Source: Arxiv · Venue: NEURIPS 25 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a benchmark for evaluating LLM-based WebAgents on long-context reasoning, where agents must retrieve and apply information from prior interaction histories spanning 25k–150k tokens. All four tested frontier models (Claude-3.7, GPT-4.1, Llama-4, o4-mini) degrade catastrophically, dropping from 40–50% to <10% task success as context length grows. A proposed implicit RAG (iRAG) mitigation offers modest gains but does not resolve the fundamental failure.

Problem

Existing WebAgent benchmarks do not evaluate performance under realistic long-term, multi-session interaction histories. Prior work (RealWebAssist, PrefEval) either caps context at short lengths or simulates QA-style retrieval rather than action-taking trajectories. There is no systematic study of how action-taking agents fail as their interaction history accumulates irrelevant noise across hundreds of steps.

Method

The authors construct a benchmark using 56 curated executable tasks from WebCanvas, each split into two sequentially dependent subtasks (A1, A2). A2 is deliberately under-specified—it references an attribute established in A1—forcing the agent to retrieve prior context. Between A1 and A2 execution, irrelevant task trajectories (each containing thought/action/reflection/URL per step) are injected to pad the context to absolute lengths of 25k, 50k, 75k, 100k, 125k, or 150k tokens. Agents are evaluated using rule-based key-step completion (from WebCanvas) rather than LLM-judge to minimize evaluation variance. The iRAG approach adds a dedicated pre-step where the agent generates a task-relevant summary from the noisy history, which is then appended to the context before normal execution.

Key Contributions

  • A long-context WebAgent benchmark with sequentially dependent subtasks and controllable noise injection up to 150k tokens.
  • Comprehensive evaluation of Claude-3.7, GPT-4.1, Llama-4-17B Maverick, and o4-mini across six context lengths.
  • Taxonomy of failure modes: loops (step-limit exhaustion via repeated actions), inefficient progress (step-limit without loop), false ends (premature termination), and other errors.
  • Retrieval performance analysis decomposing task success into retrieval success/failure Ă— task success/failure quadrants.
  • An implicit RAG (iRAG) mitigation that generates a task-focused summary to assist in-context retrieval.

Results

  • Baseline degradation: Success rates fall from ~40–50% (no noise) to <10% at 150k tokens across all models.
  • o4-mini vs GPT-4.1 at 150k: 21.2% ± 3.1% vs 2.7% ± 1.1%—reasoning models substantially outperform standard models in long-context settings.
  • Dominant failure modes: GPT-4.1 loops 44.3% of the time; o4-mini loops 43.2%; Claude-3.7 loops 16.7% but suffers more from inefficient progress (35.0%).
  • Retrieval analysis: No model achieves task success when retrieval fails (0% TS, RF across all models); retrieval success does not guarantee task success (Claude-3.7: 25.1% TS given RS, GPT-4.1: 7.0%).
  • iRAG: Improves performance for Claude-3.7, GPT-4.1, and o4-mini at 150k; GPT-4.1 benefits most from iRAG in retrieval counts.
  • Efficiency ratio: Near-human step efficiency at no-noise; sharply drops as context length increases, corroborating step-limit error dominance.

Limitations

  • Small benchmark size: 56 tasks (curated from 543 due to live-web executability decay), limiting statistical power.
  • Single experimental run takes ~16 days due to API rate limits and long-context latency.
  • Subtask splits may be artificial; many WebCanvas tasks are atomic and yield unnatural A1/A2 splits.
  • Live Internet is non-deterministic (dynamic page content, anti-bot measures) and LLMs are non-deterministic even at temperature=0, precluding exact reproducibility.
  • Llama-4 was evaluated only up to 75k tokens due to context length constraints.
  • iRAG generates summaries via the same LLM being evaluated, so it inherits the model's own retrieval weaknesses.

Relevance to Agentic AI / LLM Agents

This work directly quantifies a critical, undercharacterized failure mode for deployed LLM agents: the inability to maintain task coherence across realistic multi-session interaction histories. The finding that retrieval failure—not execution failure—is the dominant bottleneck (agents often succeed if they retrieve correctly, but mostly do not retrieve) establishes a concrete diagnostic target for memory architecture research. The iRAG result suggests decomposing context processing into a dedicated retrieval sub-step is a viable direction but insufficient alone, motivating future work on external memory, context compression, and episodic recall for agents. The benchmark design—injecting orthogonal noise trajectories at controlled token lengths—is a reusable methodology for stress-testing any long-horizon agent system.