Skip to content

Thinking vs. Doing: Improving Agent Reasoning by Scaling Test-Time Interaction

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes scaling the number of environment interaction steps at test time as a new, orthogonal axis to per-step reasoning compute for LLM agents. They introduce TTI, a curriculum-based online RL method that trains agents to adaptively extend their interaction horizon, achieving state-of-the-art among open-source, open-data web agents on WebVoyager (64.8%) and WebArena (26.1%) using Gemma 3 12B.

Problem

Existing test-time scaling for agents focuses on generating longer reasoning traces per step (CoT, budget forcing, best-of-n), but this cannot acquire new information from the environment. Reactive policies optimized for a single "best action" per step are sub-optimal in partially observable or non-stationary environments where information-seeking, backtracking, and re-planning within a rollout are necessary. Training agents at fixed long horizons naïvely causes noisy credit assignment and aimless exploration, preventing effective learning.

Method

TTI frames agent training as filtered online behavior cloning (REINFORCE with binary rewards, retaining only successful trajectories) with a multiplicative curriculum over the maximum interaction horizon h. Starting from h_min = 10, the allowed rollout length grows as h_i = clip(h_min · i, h_max) per iteration, forcing the agent to first master short, exploitative paths before learning longer exploratory behaviors. Training uses 128K synthetic tasks auto-generated by the base model itself (Gemma 3 12B) browsing real websites, with a Gemma 3 27B prompted verifier labeling successful trajectories (88.9% accuracy vs. ground truth). Rollouts are stored in a recency-weighted replay buffer; vLLM is used for sampling and DeepSpeed ZeRO-3 for training on H100s.

Key Contributions

  • Formalizes interaction scaling (increasing rollout horizon h) as a distinct, complementary axis to per-step compute scaling.
  • Demonstrates via controlled prompting experiments that interaction scaling outperforms budget forcing and best-of-n under equal token budgets on WebArena.
  • Proposes TTI, a curriculum-based online RL algorithm that grows interaction horizon multiplicatively to avoid premature overfitting and noisy credit assignment from fixed long horizons.
  • Achieves SOTA among fully open-source, open-data web agents: 64.8% on WebVoyager and 26.1% on WebArena with Gemma 3 12B.
  • Shows TTI learns to adaptively trade off exploration vs. exploitation: longer rollouts in complex tasks, efficient direct paths in simple ones; per-step reasoning length actually decreases as interaction length grows.

Results

  • Prompting baseline (WebArena, 62-task subset): "check-again" re-checking raises SR from 23.81% (CoT only) to ≥28%; spending the same token budget on budget forcing yields <3% gain.
  • WebVoyager (427 tasks): TTI 64.8% vs. fixed h=10 59.1% (+5.7%), fixed h=30 45.2% (+19.6%), zero-shot Gemma 3 12B 55.8% (+9%). Best open-source open-data model; UI-TARS-1.5 (proprietary data) reaches 84.8%.
  • WebArena (812 tasks): TTI 26.1% vs. zero-shot 18.3% (+7.8%), fixed h=10 23.8% (+2.3%), fixed h=30 19.0% (+7.1%); best fully open-source self-improvement agent. Top proprietary agent (IBM CUGA) reaches 61.7%.
  • Multiplicative curriculum (32.25%) outperforms additive (29.50%) in ablation on WebArena subset after 10 iterations.
  • Applying inference-time re-checking on top of TTI checkpoints yields additional gains, especially early in training.

Limitations

  • Experiments restricted to web navigation; generalization to robotics, open-world games, or other interactive domains untested.
  • Increased interaction steps raise inference and training compute costs; efficient long-context handling is an open problem.
  • Training uses only filtered behavior cloning (online STaR); more advanced RL (PPO, GRPO) not explored.
  • Full benchmark evaluated only once per setting; variance from policy, environment, and evaluator stochasticity is not quantified.
  • TTI agents exhibit failure modes: over-reliance on search resets as a universal fallback rather than domain-specific recovery, and limited self-verification (agent identifies a mismatch with task goal but still submits wrong answer).

Relevance to Agentic AI / LLM Agents

This work directly challenges the dominant paradigm of test-time scaling (longer CoT per step) by showing that acting more in the environment is a more compute-efficient strategy for partially observable interactive tasks — a core property of real-world agentic settings. The curriculum-based interaction horizon schedule provides a practical RL recipe for training agents that can switch between exploration and exploitation within a single rollout, addressing a key gap in current post-training methods. The self-improvement loop (base model generates training data, trains itself) is directly relevant to scalable agent training without proprietary supervision. For researchers tracking agentic AI, TTI offers both a conceptual reframing (interaction steps as a scaling axis) and an empirical baseline on WebVoyager/WebArena that is fully reproducible with open models and data.