Skip to content

VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applications

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Versatile real-world interactive benchmark covering many agent capability dimensions

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VitaBench is a benchmark for LLM agents grounded in real-world life-serving applications (food delivery, in-store consumption, online travel), comprising 66 tools across 400 tasks with multi-turn user interaction. It introduces a three-dimensional complexity framework (reasoning, tool, interaction) and a rubric-based sliding window evaluator. Even the best model (o3) achieves only 30.0% Avg@4 on cross-scenario tasks.

Problem

Existing agent benchmarks fail to capture the joint complexity of real-world deployments: early benchmarks test single-turn function-calling accuracy; more recent ones (\(\tau\)-bench, \(\tau^2\)-bench) impose rigid domain-specific policies and constrained action spaces; none simultaneously stress all three axes—information-dense reasoning, intricate inter-tool dependencies, and dynamic stochastic user interaction—nor adequately treat users as uncertain environmental components.

Method

VitaBench formalizes agent tasks as POMDPs \((U, S, A, O, T, r)_e\) where \(S = S_{db} \otimes S_{user}\) and \(T\) decomposes into deterministic API transitions \(T_{db}\) (Python functions) and stochastic user transitions \(T_{user}\) (LLM-simulated). Task complexity is quantified along three axes:

  • Reasoning complexity \(C_{reason}\): observation-space entropy \(H(O)\) and partial observability \(\eta = 1 - |O|/|S|\); operationalized via large databases with target/distractor items and composite multi-requirement instructions.
  • Tool complexity \(C_{tool}\): toolset modeled as a directed graph \(G=(V,E)\) with \(|V|=66\) tools; complexity metrics are edge density \(\rho = |E|/(|V|(|V|-1))\) and task-subgraph coverage ratio \(|V_{task}|/|V|\); cross-scenario tasks expand \(A\) across domains.
  • Interaction complexity \(C_{interact}\): LLM-simulated users with per-task profiles (age, dietary restrictions, persona) and behavioral attributes (patience, goal ambiguity, cooperation level) that evolve dynamically across 50–100 turns.

Policy rules are encoded structurally as tool pre/post-conditions rather than verbose policy documents, enabling autonomous agent exploration. A rubric-based sliding window evaluator divides each trajectory into overlapping windows of \(w\) consecutive turns (overlap \(\delta\) turns), extracts rubric-relevant evidence per window, and maintains a persistent satisfaction state vector \(s \in \{0,1\}^k\); success requires \(\sum_j s_j = k\) (all-or-nothing). The evaluator is implemented with Claude-3.7-Sonnet; inter-rater agreement Cohen's \(\kappa \geq 0.81\).

Key Contributions

  • VitaBench benchmark: 400 tasks (100 cross-scenario + 300 single-scenario) across delivery, in-store, and OTA domains with 66 tools, rich databases (up to 9,693 products, 1,437 service providers), and authentic user profiles.
  • A formal three-dimensional task complexity framework (\(C_{reason}\), \(C_{tool}\), \(C_{interact}\)) grounded in POMDP theory.
  • Policy-free tool dependency graph design that encodes domain rules structurally, enabling flexible cross-domain composition without verbose policy documents.
  • Rubric-based sliding window evaluator capable of handling long trajectories (50–100 turns) with multiple valid solution paths and intermediate state tracking.
  • Systematic failure analysis identifying reasoning errors (61.8%), tool usage errors (21.1%), and interaction management failures (7.9%) as dominant failure modes.

Results

  • Cross-scenario (main): Best model (o3, thinking) achieves 30.0% Avg@4; 61.0% Pass@4; 6.0% Pass\(\hat{\,}\)4.
  • Single-scenario best: Claude-4.1-Opus (thinking) reaches ~53.5% Avg@4 on OTA; best single-domain scores exceed 50% but remain well below ceiling.
  • Thinking vs. non-thinking: Thinking models outperform non-thinking counterparts (e.g., Claude-4.1-Opus: 21.8% → 29.0% cross-scenario Avg@4; GLM-4.5: 20.0% → 22.8%); thinking models also use fewer turns on average (61.1 vs. 69.9).
  • Exploration instability: Pass@4 improves substantially with sampling (e.g., o3: 61.0%), but Pass\(\hat{\,}\)4 near zero for most models even at \(k=32\), indicating fundamental consistency failures.
  • Failure distribution: 61.8% reasoning errors, 21.1% tool usage errors, 7.9% interaction management failures.
  • Baselines compared: DeepSeek-V3-0324 (3.8% cross Avg@4), GPT-4.1 (13.8%), Claude-4-Sonnet non-thinking (21.3%), o3 high (30.0%).

Limitations

  • Tasks are limited to three Chinese life-serving application domains (delivery, in-store, OTA); generalization to other sectors is untested.
  • The user simulator uses GPT-4.1 as backbone, introducing its own biases and potential inconsistencies that could affect reproducibility.
  • All-or-nothing success scoring may obscure partial progress; rubric granularity is manually designed and domain-specific.
  • Evaluation with Claude-3.7-Sonnet as judge may introduce model-specific evaluation biases.
  • The minimal agent setup (no RAG, no memory modules, no reflection loops) means results reflect base LLM capabilities, not the current frontier of engineered agentic systems.
  • Tasks capped at a finite turn limit; real applications may require even longer horizons.

Relevance to Agentic AI / LLM Agents

VitaBench directly advances the agentic AI benchmarking frontier by jointly stressing all three axes that matter in deployment—long-horizon reasoning under partial observability, navigation of a 66-tool dependency graph without policy scaffolding, and multi-turn dialogue with stochastic simulated users—exposing that current SOTA models cap out at 30% on realistic cross-domain tasks. The policy-free tool graph design is a methodological contribution: by encoding domain constraints structurally rather than textually, it shifts agent evaluation from instruction-following toward autonomous exploration, which is the harder and more relevant capability. The rubric-based sliding window evaluator addresses a core infrastructure gap for long-trajectory agentic evaluation, providing dense intermediate signals usable for RL fine-tuning. The instability finding (Pass\(\hat{\,}\)4 ≈ 0 even for top models) highlights that reliability under repeated execution—not just peak performance—is the critical unsolved problem for production agent deployment.