Skip to content

OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

🕒 Published (v1): 2026-07-16 13:38 UTC · Source: Arxiv · link

Why this paper was selected

Broad general-agent benchmark spanning tool use + multi-step tasks; key eval reference for harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OmniaBench is a large-scale agent benchmark spanning 90 level-1 and 354 level-2 domains (consumer, business, enterprise) with 1,431 tasks derived from real app-store and product-document sources. It evaluates general LLM agents across ten capability dimensions and eight atomic difficulty factors using stateful, POMDP-formulated environments. Even frontier models like Claude-Sonnet-5 and GPT-5.6-Sol achieve only ~58 Pass@1, indicating substantial headroom.

Problem

Existing agent benchmarks focus on narrow scenario sets, single tool ecosystems, or restricted interaction formats, creating a gap between benchmark coverage and the heterogeneous reality of general-agent deployment. No prior benchmark simultaneously covers diverse domain taxonomies, multi-turn interaction, persona constraints, fine-grained capability decomposition, and compositional difficulty analysis.

Method

OmniaBench formalizes each task as a POMDP \(P = (S, O, A, T, \Omega, \rho_0)\) where an agent policy \(\pi_\theta\) acts on observation histories \(h_t = (o_0, a_0, \ldots, o_t)\) rather than the latent state \(s_t\).

Taxonomy: A three-view hierarchy — domain (ToC/ToB/ToE, 90 L1 / 354 L2), capability (10 dimensions), and atomic difficulty (8 factors) — is seeded from app stores, PRDs, web-search agents, and human refinement.

Environment construction: Per level-2 domain, a web agent collects domain knowledge; LLM-assisted synthesis infers entities, state variables, and constraints, which are compiled into Python environment classes with tool schemas, executable function implementations, and sandbox adapters for filesystem/code tasks.

Task synthesis via four routes: 1. DAG — tool dependency graphs are sampled to form executable tool chains; tasks and reference trajectories are generated conditioned on environment states (multi-turn). 2. DAG-S — single-turn reformulations of DAG tasks via query refinement. 3. Solver — selection/scheduling/optimization tasks using solver-guided synthesis. 4. Program — complex procedural reasoning with branching, iteration, and debugging.

Evaluation: Rubric-based scoring (weighted 1–3 per item, binary per item) for DAG/DAG-S/Solver; VerifyCode binary verifiers for Program tasks. Multi-turn tasks use persona-grounded user simulators with configurable communication styles and disclosure strategies. A challenging subset of 644 tasks retains the longest-chain DAG task per L2 domain plus most Solver and Program instances.

Key Contributions

  • OmniaBench benchmark: 1,431 tasks (644 challenging subset) across 90 L1 / 354 L2 real-world domains — substantially broader than prior benchmarks (next largest: 44 domains in GDPval).
  • Ten-dimensional capability taxonomy: Task Understanding, Information Gathering, Planning & Decision Making, State Management, Tool Use, Code & Programmatic Operations, Data Analysis, Office & Document Handling, Interactive Collaboration, Reliability & Safety.
  • Compositional atomic difficulty framework: 8 challenge factors spanning user intent/persona, goal decomposition, environment state changes, tool execution, multi-turn interaction, error recovery, and result verification.
  • Four complementary data synthesis routes (DAG, DAG-S, Solver, Program) enabling diverse task forms within unified evaluation.
  • Systematic multi-model evaluation of 20+ closed- and open-source frontier models with per-domain, per-capability, per-difficulty, and error-pattern breakdowns.

Results

  • Claude-Sonnet-5: 58.54 overall Pass@1 (best closed-source).
  • GPT-5.6-Sol: 57.14 overall Pass@1.
  • GLM-5.2: 53.xx (third in leaderboard, based on figure).
  • Performance drops substantially for open-source models, with several scoring in the 20–40 range.
  • Domain breakdown (ToB/ToC/ToE) and capability breakdown (e.g., Planning & Decision Making, State Management) reveal persistent weaknesses — planning, constraint maintenance, and adaptive correction are consistently poor.
  • Interaction and tool-step metrics show models degrade as task horizon and turn count increase.
  • Evaluation was conducted June 29 – July 15, 2026 using provider APIs; no single model exceeds 60 Pass@1.

Limitations

  • Evaluation cost is non-trivial; the 644-task challenging subset is introduced specifically to reduce cost, but the full 1,431-task set may still see limited adoption.
  • Post-release contamination risk for the full set motivated the challenging subset, but the subset itself may eventually be contaminated.
  • Tasks are generated by LLM-assisted pipelines with human curation, so residual synthesis artifacts or ambiguous rubrics may persist despite validation passes.
  • The benchmark does not cover multimodal (vision) grounding — task inputs are text/tool-response only.
  • Rubric-based scoring for open-ended items relies on LLM judgment (not reported here in detail), introducing inter-rater variance.
  • User simulator fidelity in multi-turn tasks is bounded by the simulator model's quality and may not capture all real-user behaviors.

Relevance to Agentic AI / LLM Agents

OmniaBench directly addresses a core bottleneck in agentic AI research: the absence of a broad, diagnostic benchmark that covers heterogeneous real-world scenarios rather than narrow tool ecosystems. Its POMDP formulation, persona-grounded multi-turn evaluation, and ten-dimensional capability decomposition make it a strong diagnostic instrument for identifying where current LLM agents break — particularly in planning, state tracking, and error recovery. The finding that even the best frontier model scores only ~58.5 on tasks derived from everyday app-store workflows underscores how far general agents remain from practical deployment. For researchers building or evaluating agentic systems, OmniaBench provides a richer signal than single-domain benchmarks and highlights the interplay between tool diversity, multi-turn interaction, and task complexity.