Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Dynamic, asynchronous benchmark where environments evolve independently of agent

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Gaia2 is a benchmark of 1,120 human-annotated scenarios that evaluates LLM agents in asynchronous, event-driven smartphone-like environments where time advances independently of agent actions. It introduces a write-action verifier with 0.98 agreement enabling fine-grained, action-level evaluation directly usable for RLVR training. Frontier models top out at 42% pass@1 (GPT-5 high), with no model dominating across all capability dimensions.

Problem¶

Existing LLM agent benchmarks (GAIA, SWE-bench, WebArena, AppWorld) are synchronous and agent-driven: the environment only changes when the agent acts, and evaluation typically checks only final outputs. This leaves untested the core challenges of real deployment—handling asynchronous events, operating under temporal constraints, adapting to noise, resolving ambiguity, and coordinating with other agents. There is also no unified benchmark that is simultaneously verifiable at the action level and directly usable for RLVR.

Method¶

Gaia2 is built on the Agents Research Environments (ARE) platform, which provides event-based, time-driven simulations that run asynchronously from the agent. Key design elements:

Mobile environment: 12 apps (Messages, Emails, Calendar, Contacts, Shopping, etc.), 101 tools, 10 unique "universes" each containing 400K–800K tokens of synthetic but internally consistent content seeded from PersonaHub personas.
Asynchronous time: Model generation time directly consumes simulated time; events (e.g., incoming messages) fire on absolute or relative timestamps during the agent's reasoning, making temporal awareness evaluable.
Scenario DAGs: Each scenario specifies an initial state, a DAG of write-action ground truth and environment events, and a verification method. Scenarios are annotated by humans and validated through multiple independent rounds plus automated guardrails.
ARE Verifier: Evaluates every state-changing write action (read actions are unrestricted) against oracle annotations across four dimensions: (i) Consistency (exact argument matching for rigid fields, LLM-rubric for flexible text); (ii) Causality (dependency DAG ordering); (iii) Timing (tolerance windows); (iv) Completeness (all oracle writes must be matched). Order-agnostic for independent goals.
Seven capability splits: Execution, Search, Ambiguity, Adaptability, Time, Noise (injected tool anomalies + distractors), Agent2Agent (apps replaced by sub-agents requiring explicit coordination via messaging).
Scaffold: Model-agnostic ReAct loop outputting one JSON tool call per step, augmented with pre/post-step hooks for notification injection and termination checks.

Key Contributions¶

ARE framework: Open-source platform for building asynchronous, event-driven agent benchmarks with scalable evaluation and RL data generation; shown to faithfully reimplement \(\tau\)-bench, \(\tau^2\)-bench, GAIA, BFCL-v3, and VendingBench.
Gaia2 benchmark: 800 unique human-authored scenarios (+ 320 augmented) across 7 capability splits, the first to unify asynchronous execution, temporal reasoning, noise robustness, ambiguity resolution, and multi-agent collaboration under a single verifiable framework.
ARE Verifier: Standalone, reusable action-level verifier achieving 0.98 agreement / 0.99 precision on 450 labeled trajectories, substantially outperforming LLM-only judges (0.72 agreement); directly usable as a RLVR reward signal.
Empirical study: Evaluation of 14 proprietary and open-source models exposing fundamental trade-offs between reasoning strength, latency, robustness, and cost.

Results¶

GPT-5 (high) achieves the best overall pass@1 of 42.1%, with an ~8-point margin over Claude-4 Sonnet (34.8%) across all splits.
Kimi-K2 leads open-source models at 21% pass@1 overall (from table: ~25.8%).
Execution and Search are easiest; Ambiguity, Adaptability, and Time are hardest.
Time split: Only Gemini 2.5 Pro and Claude-4 Sonnet achieve meaningful scores; GPT-5 (high) scores 0.0 on Time despite leading overall, reflecting a latency penalty in asynchronous settings.
Noise: Most models score below 20%; GPT-5 (high) reaches 35.4%.
Agent2Agent: Collaboration augmentation benefits weaker models more than frontier systems.
All budget-scaling curves plateau, indicating that standard ReAct scaffolds and current models lack ingredients for sustained progress at higher cost budgets.
ARE Verifier: 0.98 agreement, 0.99 precision, 0.95 recall vs. LLM-only verifier at 0.72/0.53/0.83.
Parallel Tool Calling improves efficiency (wall-clock time and token usage) but not pass@1 accuracy, confirming bottlenecks are intrinsic to model capabilities.

Limitations¶

Best overall score is only 42% (GPT-5 high); the benchmark is largely unsolved, but this also means relative rankings may shift significantly as models improve.
The Mobile environment is consumer-domain-specific; generalization to desktop, enterprise, or web domains requires new environment definitions (though ARE is claimed environment-agnostic).
The ARE Verifier uses Llama-3.3-70B-Instruct as a sub-judge for flexible text fields, introducing a dependency on that model's correctness and potential bias.
Scenarios are annotated by human workers using ARE's GUI rather than native OS, making the human performance baseline artificially slow and not a clean upper bound.
The A2A split uses the same underlying model for both main and app-agents, which may not reflect realistic heterogeneous multi-agent deployments.
No adversarial or red-teaming scenarios; noise is controlled perturbation rather than adversarial manipulation.

Relevance to Agentic AI / LLM Agents¶

Gaia2 directly addresses a critical evaluation gap: most current benchmarks cannot test whether agents handle real-world asynchrony, temporal urgency, or multi-agent coordination, all of which are central to deploying practical LLM agents. The write-action verifier and RLVR-compatible reward signal make Gaia2 a training substrate, not just a leaderboard—positioning it as infrastructure for the next generation of agent post-training. The budget-scaling plateau finding is particularly significant: it empirically shows that simply scaling test-time compute with standard ReAct scaffolds hits a ceiling, motivating new architectures and training regimes. The ARE framework's ability to reimplement existing benchmarks (\(\tau\)-bench, BFCL, GAIA) within a single asynchronous abstraction could serve as a unifying substrate for the agent evaluation and training community.