Skip to content

LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

🕒 Published (v1): 2026-06-15 14:42 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LabOSBench is a browser-based benchmark of 96 subtasks across eight web-simulated scientific instruments (SEM, TEM, FIB, XRD, etc.) for evaluating multimodal computer-use agents. It avoids OS-level virtualization by running entirely in a browser with Playwright, using in-page benchmark scripts and a Python coordinator to log episodes and compute metrics. Best agent (GTA1 w/ GPT-5.5) achieves 81.4% subtask-level but only 26.3% end-to-end success, revealing severe error accumulation in long-horizon workflows.

Problem

Existing computer-use benchmarks (OSWorld, WebArena) rely on heavyweight VM/Docker virtualization and evaluate general GUI tasks, providing no coverage of scientific-instrument interfaces that require procedural dependencies, feedback-driven parameter tuning, and dense domain-specific controls. Evaluating agents on physical instruments is unsafe, expensive, and non-reproducible.

Method

LabOSBench builds eight high-fidelity web-based instrument simulators (SEM, TEM, LFM, XRD, EDS, APT, FIB, SPM). Each simulator is instrumented with an in-page JavaScript harness (e.g., benchmark_xrd.js) that tracks subtask completion, control attempts, and step-level traces. A Python coordinator drives interaction via Playwright: at each step, it sends a screenshot + NL instruction to the agent, receives a GUI action (click/drag/type/scroll/wait), dispatches it via DOM events or mouse fallback, captures the new screenshot, and queries in-page state. Episodes export JSON logs conforming to instrument-specific JSON Schemas. For subtask-level evaluation, programmatic fast-forward functions (XRD_fast_forward_to_subtask(Sk)) initialize simulators to canonical pre-subtask states. Success is scored as average subtask success rate per instrument; image-quality tasks additionally report PSNR against a reference view.

Key Contributions

  • LabOSBench: a lightweight, browser-native, executable benchmark requiring no OS virtualization, with configurable CLI/env-var episode launch and JSON Schema-validated episode logs.
  • 96 subtasks across 8 instrument simulators spanning microscopy, diffraction/spectroscopy, and microfabrication, decomposed into five operational categories with three difficulty tiers (Easy/Medium/Hard).
  • Dual evaluation modes: subtask-level (isolated, fast-forwarded initialization) and full end-to-end (from cold start within a step budget), distinguishing local grounding failures from long-horizon accumulation failures.
  • Systematic evaluation of 12 agents/frameworks spanning general VLMs, specialized GUI models (UI-TARS, GUI-Owl), and agentic frameworks (GTA1, VLAA-GUI, Hippo Agent).

Results

  • Best overall: GTA1 w/ GPT-5.5 — 81.4% subtask-level average across eight instruments.
  • Best general model: Seed-1.6 — 76.3% subtask average.
  • Specialized GUI models (UI-TARS-1.5-7B: 65.9%, GUI-Owl-7B: 64.0%) underperform top general VLMs despite GUI-specific training.
  • VLAA-GUI w/ Opus-4.5 (45.6%) underperforms even standalone Opus-4.5 (62.0%), showing agent loops can hurt without domain-valid recovery.
  • End-to-end (GPT-5.5 only, 10 runs/workflow): 26.3% average; 0% on SEM, SPM, TEM, LFM, FIB; 60–80% only on short-workflow instruments (EDS 70%, APT 80%, XRD 60%).
  • Hardest instruments: FIB (longest workflow, lowest average model score), LFM (large human–model gap due to feedback-driven optical adjustment).
  • Human baseline: 92.4% subtask average; human–model gap largest on LFM and FIB.
  • SEM focus task: GPT-5.5 achieves highest PSNR and focus success rate among all models.

Limitations

  • Simulators do not capture hardware latency, calibration uncertainty, real safety constraints, or physical failure modes.
  • Coverage limited to microscopy, spectroscopy, diffraction, and tomography; excludes wet-lab protocols, robotic manipulation, chemical synthesis, and multi-instrument planning.
  • Evaluation relies on screenshots and logged simulator state; some scientific decisions require richer multimodal sensor signals or domain knowledge unavailable in current environments.
  • Not safe or intended for deployment on physical laboratory hardware without human supervision, permission control, and safety interlocks.

Relevance to Harnesses / Meta-Harnesses

LabOSBench is itself a benchmark harness: a thin Python coordinator that wraps browser-driven simulators, injects per-instrument JavaScript evaluation hooks, exports structured JSON episode logs via a defined schema contract, and dispatches actions through a unified Playwright driver — exactly the architecture pattern of agent evaluation meta-harnesses. The in-page benchmark_*.js scripts and fast-forward initialization functions are a domain-specific implementation of the harness abstraction layer (configure state → observe → act → score). The dual subtask/end-to-end evaluation modes demonstrate how harness design choices (isolated vs. chained initialization) directly determine what failure modes are diagnosable, a key design concern in building meta-harnesses for long-horizon agentic systems.