Skip to content

AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

🕒 Published (v1): 2026-07-06 14:56 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

ACL 2026; de-idealized real-world agent benchmark; addresses gap between lab and production evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentGym2 is a benchmark that stress-tests LLM agents in de-idealized real-world environments: tasks require discovering undisclosed tools, handling noisy/underspecified inputs, and completing end-to-end procedures without pre-packaged scaffolding. Evaluating 15 models, even GPT-5 reaches only ~46% and the best open-source model ~32%, exposing a large gap between current agents and production readiness.

Problem

Existing agent benchmarks (AgentBench, AgentGym, DA-Code, Ď„-Bench) evaluate under idealized conditions: pre-selected domain-specific tools are provided upfront, intermediate steps (e.g., data cleaning) are pre-solved, and inputs are clean and fully specified. This systematically understates real deployment difficulty, where agents must discover tools dynamically, interpret vague user intent, and reject misleading information.

Method

AgentGym2 builds a layered, modular evaluation framework with three scenario types:

  1. Complex Tool Use (182 tasks): agents are given only a composable base toolbox (27 actions across web browsing, retrieval, file processing, multimodal understanding, code execution) and must discover additional online APIs/services and compose novel workflows. Two sub-types: tool discovery (57 instances) and tool composition (125 instances).

  2. End-to-End Data Analysis (57 tasks): tasks sourced from GitHub/Kaggle require the full pipeline—EDA, deduplication/normalization, and analysis—on raw, uncleaned data, with the expected output being a specific numerical answer.

  3. Deep Search (198 tasks): queries contain injected ambiguity or memory-induced noise (incorrect attributes). Sub-types: ambiguity-only (114 instances, multi-hop Wikipedia-derived) and ambiguity-with-bias (84 instances, sourced from Reddit/Xiaohongshu with noisy hedged attributes). Agents must denoise queries to retrieve the correct unique answer.

Tasks are sourced from GitHub, Reddit, and Kaggle; verified by a cascading pipeline (model-based + human review) for solvability, uniqueness of answer, and clarity. Evaluation uses short-string/numeric reference answers judged by Qwen3-235B-A22B-Instruct via LLM semantic equivalence. Each task runs 3Ă— at temperature 0.6, reporting Avg@3 and Pass@3. Agents use ReAct-style interaction capped at 100 rounds; parallel asynchronous tool invocation is supported.

Key Contributions

  • First benchmark combining non-simulated environments, realistic composable action space, tool discovery (no upfront tool list), E2E data analysis with cleaning, and injected distractors—simultaneously.
  • A modular architecture with runtime isolation (sandboxed code execution, isolated browser instances, file management to prevent read-write conflicts) enabling reproducible parallel evaluation at scale.
  • Full internet access for agents (not pre-restricted websites), forcing true open-ended tool acquisition.
  • Controlled noise injection methodology for deep search: two perturbation types (ambiguity-only, ambiguity-with-bias) to quantify robustness.
  • Comprehensive scaling analysis across model size, interaction rounds, and API cost.

Results

  • GPT-5: 46.15 Avg@3 overall (best proprietary; 48.72 on Complex Tool Use, 39.18 on Data Analysis, 45.80 on Deep Search).
  • Claude-4.5-Sonnet: 37.17 Avg@3 overall; notably strong on Data Analysis (39.77 Avg@3) relative to peers.
  • Gemini-2.5-Pro: 20.67 Avg@3, substantially below the other two proprietary models.
  • Nex-N1-671B (best open-source): 32.19 Avg@3; outperforms its base (DeepSeek-V3.1, 21.66) by ~10.5 points.
  • Nex-N1-32B: 15.71 Avg@3; outperforms its base (Qwen3-32B non-thinking, 6.55) by ~9.16 points.
  • Data Analysis is the hardest scenario: GLM-4.6 drops 13.97 points from Complex Tool Use (22.16) to Data Analysis (8.19).
  • Model size scales: Qwen3 non-thinking: 8B→32B→235B yields 8.70→13.50→25.40 Avg@3.
  • Interaction rounds do not scale: no improvement trend with more turns; thinking-mode models use fewer turns but underperform instruct variants (Qwen3-235B-Think lags instruct by 5.31 points).
  • Pass@k vs. Pass^k: Nex-N1-671B gains ~30% absolute on Pass@k with larger \(k\), but Pass^k collapses sharply, exposing low per-trajectory reliability.

Limitations

  • Only 437 total tasks; Data Analysis is particularly small (57 instances), limiting statistical power.
  • Three scenario types may not cover all real-world agentic use cases (e.g., no embodied/robotic tasks, no long-horizon software engineering).
  • LLM-based judge (Qwen3-235B) for answer equivalence introduces potential judge bias; human–LLM agreement analysis is mentioned but only partially reported in the provided text.
  • ReAct-style agent framing is fixed; results may not transfer to other architectures (e.g., plan-then-execute or reflection-based agents).
  • Internet-connected evaluation introduces reproducibility risk as external web content and APIs can change over time.
  • Cost of evaluation is significant; the \(512–\)1024 regime per run limits community accessibility.

Relevance to Agentic AI / LLM Agents

AgentGym2 directly targets the benchmark gap that inflates perceived agent capability: by requiring tool discovery (no upfront tool list), noise robustness (injected distractors), and end-to-end execution (no pre-solved intermediate steps), it surfaces failure modes—insufficient exploration, hallucination under ambiguity, brittle data pipelines—that existing idealized benchmarks mask. The finding that agentic post-training (Nex-N1 series) yields ~9–10 point gains over base models suggests that domain-targeted RL or RLHF on agentic trajectories is a tractable path forward. The pass^k collapse is a key signal for agent deployment: even when a model can solve a task, per-run reliability remains the binding constraint for production use.