Skip to content

SCUBA: Salesforce Computer Use Benchmark

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Real-world CRM computer-use benchmark from Salesforce; 300 task instances from user interviews

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SCUBA is a 300-task benchmark for evaluating computer-use and browser-use agents on enterprise CRM workflows within the live Salesforce platform. It introduces milestone-based process rewards alongside binary success, plus latency and cost metrics. A large gap exists between open-source GUI agents (<5% success) and the best closed-source agentic frameworks (up to 39–50% with demonstrations).

Problem

Existing computer-use benchmarks (WebArena, OSWorld) and enterprise benchmarks (WorkArena, CRMArena) do not capture the full complexity of enterprise-grade, read-and-write CRM workflows: WorkArena is limited to ServiceNow service tasks; CRMArena evaluates tool-use/retrieval agents that never mutate data; none provide process rewards, human demonstrations, or latency/cost metrics. There is no realistic benchmark spanning platform administration, sales, and service workflows in a live enterprise environment.

Method

SCUBA runs agents against Salesforce sandbox developer orgs (UI identical to production). Tasks are constructed in four phases: (1) 60 workflow templates sourced from real-user interviews, each paired with a Trailhead knowledge article and a difficulty label; (2) five parameterized query instances per template, paraphrased by GPT-5 and verified by human authors; (3) rule-based evaluators using Salesforce Metadata and Tooling APIs, emitting a binary success flag plus weighted milestone scores (process rewards) with failure reasons; (4) full human annotation for quality control and trajectory collection. Evaluation supports asynchronous parallel execution (browser sessions or Docker-containerized Ubuntu desktops), completing in under 90 minutes. Nine agents are tested under zero-shot and demonstration-augmented settings across two observation/action paradigms: browser-use (SOM + DOM, 19 actions) and computer-use (screenshot, 15 actions).

Key Contributions

  • 300 enterprise CRM task instances (60 templates × 5 instances) grounded in real Salesforce user interviews across admin, sales, and service personas.
  • Multi-dimensional evaluation harness: binary task success, milestone/process-reward scores with rubrics, per-run latency, token count, and dollar cost.
  • Asynchronous parallel evaluation infrastructure enabling full benchmark completion in <90 minutes.
  • Human-annotated demonstration trajectories that can be injected to improve agent success rates while reducing cost and latency.
  • Comprehensive baseline across 9 agents (4 browser-use: GPT-5, o3, Gemini-2.5-Pro, Claude-4-sonnet; 5 computer-use families) under both zero-shot and augmented settings.

Results

Zero-shot setting: - Best browser-use agent (GPT-5): 51.33% success, milestone score 0.73. - Best computer-use agentic framework (Agent-S2.5 w/ GPT-5): 39.00% success, milestone score 0.58. - Best single computer-use model (Claude-4-sonnet computer): 27.00% success. - Open-source native GUI agents on strong OSWorld performers: <5% success (UI-TARS-1.5-7B: 2.67%, OpenCUA-7B: 0.67%, GUI-Owl-7B: 1.00%).

Demonstration-augmented setting: - Best browser-use (GPT-5): 53.85% success; Gemini-2.5-Pro jumps from 31% to 46.15%. - Claude-4-sonnet computer-use: 47.69% (+20.69 pp over zero-shot). - Agent-S2.5: 40.38% (marginal gain). - Demonstrations reduce average time by 13% and costs by 16% for applicable agents.

Limitations

  • Benchmark is restricted to the Salesforce platform; generalizability to other enterprise software (SAP, Dynamics, ServiceNow) is unknown.
  • Task reset is task-level (reverting only agent-modified configs), not a full org reset; unexpected side effects between tasks could corrupt state.
  • 300 instances across 60 templates is relatively small; template diversity may not capture the full long-tail of enterprise workflows.
  • Evaluation bottleneck for API-based models is the provider's rate limits, not the benchmark infrastructure.
  • Human demonstration trajectories were collected under controlled conditions and may not reflect the variance of real user behavior.
  • The paper text is truncated; explicit stated limitations in a dedicated section may exist beyond what is visible here.

Relevance to Agentic AI / LLM Agents

SCUBA directly measures the gap between generic desktop-agent benchmarks and real enterprise deployment, making it a critical calibration point for anyone building or evaluating LLM agents intended for business automation. The drastic performance collapse of OSWorld-leading open-source models (<5% on SCUBA vs. strong OSWorld scores) demonstrates severe distribution shift in enterprise GUI complexity and underscores that benchmark performance does not transfer to production. The milestone/process-reward evaluation scheme is methodologically significant: it allows partial-credit attribution of agent failure to specific subtask steps, which is more informative for agent improvement than binary success alone. The demonstration-augmentation results—higher success at lower cost and latency—validate a practical deployment pattern (few-shot trajectory injection) that is directly applicable to LLM agent pipelines targeting enterprise SaaS.