Skip to content

Environment Scaling for Interactive Agentic Experience Collection: A Survey

🕒 Published (v1): 2025-11-12 12:56 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This survey formalizes the Generation-Execution-Feedback (GEF) loop as the core paradigm for training LLM agents through environmental interaction, then systematically taxonomizes methods for scaling environments across all three loop stages. It argues that environments are not passive containers but active producers of experiential data whose scaling along complexity, realism, and feedback richness is the central bottleneck for agent capability advancement.

Problem

Static, human-annotated datasets for SFT are costly, bounded by human-level knowledge, and lack dynamism—making them insufficient for training agents toward adaptive behavior and long-horizon decision-making. Despite growing interest in RL-based agent training, there is no unified framework organizing the fragmented literature on environment scaling across task generation, execution, and feedback.

Method

The paper proposes the GEF loop (Generation → Execution → Feedback) as a unifying lens and constructs a three-stage taxonomy of environment scaling dimensions:

  • Stage 1 – Task Generation: Complexity scaling (static structural intricacy: single-step → hierarchical/graph-based); Dynamic scaling (curriculum-based adaptive difficulty, challenger-solver co-evolution); Diversity scaling (heterogeneous environments to prevent overfitting).
  • Stage 2 – Task Execution: Interactivity scaling (real API/GUI execution, parallel/depth tool use, context management via MCP); Realism scaling (live executable environments, offline database snapshots as cost-preserving proxies, async multi-agent social dynamics via MQTT/decoupled clocks).
  • Stage 3 – Feedback: Density (trajectory-level vs. step-level process rewards); Granularity (rubrics-as-rewards decomposing tasks into human-interpretable checklists); Automation (LLM-as-Judge, generative reward models, agentic verifiers); Objectivity (extending RLVR to non-verifiable domains via pairwise or hybrid hard+soft checks); Robustness (soft probabilistic rewards, overseer-based anti-hacking, async retry infrastructure).

Representative systems (AgentGen, AgentGym, R-Zero, RandomWorld, ARE, MiroThinker, WebRL, OSWorld, TaskCraft, Rubicon, Trinity-RFT, etc.) are mapped to taxonomy leaves.

Key Contributions

  • Formalizes the GEF loop as the unifying paradigm for environment-centric agent training.
  • Proposes a structured, nine-dimension taxonomy spanning all three GEF stages.
  • Identifies Generator-Verifier Asymmetry as a central implementation challenge: easy-to-verify domains have hard task generation; easy-to-generate domains have hard verification.
  • Surveys implementation designs across tool-use, deep research, coding, web navigation, computer-use, and embodied domains.
  • Outlines future directions: co-evolving generators/verifiers, embedded external-tool verifiers, open-ended multi-agent/multi-cultural environments, asynchronous RL frameworks.

Results

This is a survey; no original empirical benchmarks are reported. The paper cites supporting evidence from surveyed works: - Complexity scaling (hierarchical/graph-based tasks) yields "significant performance gains" over flat task structures (Table 1 cited but not reproduced in text). - AgentGen's bidirectional difficulty adjustment and challenger-solver frameworks produce progressively harder curricula with improved reasoning. - Adaptive difficulty sliding windows (Zeng et al., 2025) yield "superior performance gains and improved learning efficiency" over fixed-difficulty baselines. - Diversity scaling: "the performance upper bound achievable across all environments increases consistently" as environment diversity grows (Zhang et al., 2025b). - MiroThinker's context pruning of weakly-related tool calls produces "consistent performance gains across various benchmarks."

Limitations

  • Survey snapshot: rapid pace of agentic research means the most recent papers may be outside coverage.
  • The environment-centric perspective admits adjacent works not explicitly designed from the environment side, complicating clean boundaries.
  • Most dynamic scaling methods are restricted to verifiable tasks; non-verifiable domains (creative writing, healthcare) remain an open problem.
  • Current RL frameworks are predominantly synchronous, creating inefficiency at scale for long-horizon tasks (context window bottlenecks).
  • Offline database proxies sacrifice realism compared to live API environments.
  • Reward robustness issues (verbosity bias, position bias, reward hacking) are identified but not solved within the survey.

Relevance to Agentic AI / LLM Agents

This survey directly addresses the data infrastructure problem for training capable LLM agents: how to generate sufficiently diverse, hard, and realistic experience without human bottlenecks. The GEF loop formalization is practically important because it makes explicit what practitioners building RL pipelines for agents (e.g., coding agents, web agents) are implicitly doing, enabling principled design choices at each stage. The Generator-Verifier Asymmetry framing is a useful conceptual tool for understanding why RLVR succeeds in math/code but struggles in open-ended domains—a core unsolved problem for general-purpose agents. The taxonomy also serves as a checklist for researchers designing new agent training environments, covering dimensions (async realism, feedback granularity, rubrics-as-rewards) that are often underspecified in individual papers.