Skip to content

MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents

🕒 Published (v1): 2026-06-15 14:08 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MyPCBench is a reproducible Linux-desktop benchmark for evaluating computer-use agents as personal assistants, seeded end-to-end from a single canonical persona (Michael Scott) across 17 simulated consumer web apps and 184 real-user-inspired tasks. The best current model, Claude Opus 4.6, solves only 55.4% of tasks fully, with steep drop-offs on cross-app and long-horizon tasks. The benchmark ships with an open-source agent harness, environment, tasks, and rubric-grading judge.

Problem

Existing computer-use and desktop-agent benchmarks (WebArena, OSWorld, etc.) run against empty desktops with minimal or generic user data, excluding tasks that require login state, personal history, or cross-application identity—precisely the tasks real personal assistants must handle. No prior benchmark seeds a coherent, cross-consistent user identity at the scale of a full personal computer.

Method

The authors build a QEMU/KVM Ubuntu 24.04 VM packaged as a Docker image hosting 17 Next.js clones of real consumer services (Gmail, Chase, DoorDash, Delta, etc.), pre-logged in and deterministically seeded from a JSON persona spec for Michael Scott (1,812 bank transactions, 2,398 emails, 679 calendar events, 10,746 browser history entries, etc.). Cross-app consistency is enforced at generation time so that a single trip leaves correlated records in every plausible app. The agent harness extends the OSWorld runner: it boots the image, restores a QEMU snapshot before each task (preventing state leakage), and drives a standard CUA ReAct loop via an OSWorld-compatible HTTP Control API (GET /screenshot, POST /execute). All six evaluated models share a uniform computer+bash tool surface. Trajectories are graded by an LLM-as-a-judge (Gemini 3.1 Flash-Lite) using per-task natural-language rubrics (mean 6.5 criteria/task, weighted) from the Odysseys format, yielding three metrics: perfect rate, rubric score (partial credit), and trajectory efficiency (rubric score per agent step).

Key Contributions

  • Reproducible, cross-app-consistent Linux desktop environment (Docker/QEMU) seeded from a single persona spec across 17 custom Next.js web app clones and a full desktop stack.
  • 184 tasks drawn from real OpenClaw community requests, each with a natural-language rubric; all tasks author-verified end-to-end on the live VM.
  • Agent harness providing a uniform computer+bash tool surface, snapshot-reset between tasks, and LLM-as-a-judge rubric grading with three complementary metrics (perfect, rubric score, trajectory efficiency).
  • Failure taxonomy across five modes (premature DONE, skipped required app, surface-error abandonment, partial artifact, hallucinated persona data) with per-family concentration analysis.
  • Two scaling analyses: performance vs. number of apps touched and vs. trajectory length.

Results

  • Claude Opus 4.6: 55.4% perfect, 81.8% rubric score (only model above 50% perfect).
  • Claude Sonnet 4.6: 39.1% perfect, 65.4% rubric score.
  • GPT-5.5: 29.3% perfect, 54.1% rubric score.
  • GPT-5.4 mini: 19.0% perfect, 48.8% rubric score.
  • Qwen 3.5 35B-A3B: 7.6% perfect, 42.5% rubric score.
  • Qwen 3.5 9B: 2.7% perfect, 7.0% rubric score.
  • At 7+ apps per task, Opus drops to 36% perfect; GPT-5.5 reaches only 4.5%; GPT-5.4 mini, Qwen 35B, and Qwen 9B reach 0%.
  • Trajectory efficiency: Opus 3.61 rubric points/step vs. Qwen 9B 0.65.
  • Pattern inference (11 tasks): Opus 82% perfect, GPT-5.5 45%.
  • Aggregation, multi-step orchestration: all non-Claude models below 16% perfect.

Limitations

  • Single canonical persona (Michael Scott); generalization to other user identities is untested.
  • 17 apps cover six SimilarWeb categories but omit many consumer domains (health, social media, smart-home, etc.).
  • 184 tasks is a relatively small suite; per-cell results are single canonical runs (no statistical confidence intervals reported).
  • LLM-as-a-judge grading introduces its own error rate, though the authors cite prior auditing against human judgments from Odysseys.
  • VM resource requirements (4 vCPUs, 8 GB RAM, ~90s boot) limit evaluation throughput.
  • Bash tool availability creates a confound: it helps Claude but causes rubric degradation in Qwen 9B, making cross-model comparisons partially tool-surface-dependent.

Relevance to Harnesses / Meta-Harnesses

MyPCBench contributes a concrete agent harness architecture: a thin OSWorld-derived runner that manages VM lifecycle (boot, snapshot restore, teardown), routes observations and actions through a provider-agnostic HTTP API, enforces a step/turn budget, and hands trajectories to an LLM-as-a-judge grader—making it a reusable harness template for evaluating any CUA-conformant agent. The uniform computer+bash tool-surface normalization layer across three provider APIs (Anthropic, OpenAI, Qwen) is a practical meta-harness pattern for multi-model evaluation. The rubric-graded, snapshot-reset evaluation loop and the failure taxonomy (premature DONE, hallucinated persona data, etc.) offer direct primitives for harness designers building personal-assistant evaluation pipelines.