"Are We Done Yet?": A Vision-Based Judge for Autonomous Task Completion of Computer Use Agents

🕒 Published (v1): 2025-11-25 08:40 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Computer Use Agents (CUAs) frequently misidentify task completion — declaring success prematurely or failing to recognize actual success. This paper introduces a VLM-based evaluation-feedback framework that judges task completion from screenshots alone and feeds natural-language rationale back to the CUA for retry, achieving up to 73% classification accuracy and 27% average relative improvement in task success rate.

Problem

CUAs lack a reliable, scalable mechanism to determine whether a task is actually complete. Script-based evaluators (e.g., OSWorld) require manually written per-task verification scripts, do not generalize, and break on interface changes. Desktop GUI environments (specifically macOS) are harder to parse than web/mobile due to the absence of structured representations like HTML trees and higher visual complexity.

Method

A three-stage zero-shot pipeline using VLMs as external judges: 1. A CUA attempts the task and produces a final screenshot. 2. A VLM receives the screenshot + task description and outputs a binary judgment (done/not done) plus a natural-language rationale. 3. If incomplete, the rationale is fed back to the CUA, which reattempts from its current state (no full restart).

The evaluator is fully decoupled from the acting agent, judging solely from visual interface state. Five VLMs were evaluated as judges (GPT-4o, Claude 3.5 Sonnet, LLaVA-v1.5-7B, InternVL 2-8B, Qwen2-VL-7B) against three CUAs (Claude Computer Use, OpenAI Operator, UI-TARS). The dataset comprises 1,260 human-labeled tasks across 42 built-in macOS applications.

Key Contributions

• A human-labeled macOS task dataset: 1,260 tasks × 42 built-in apps, spanning simple to multi-step interactions (3.4× larger than OSWorld's 369 tasks).
• A zero-shot VLM-as-judge framework for CUA task completion that requires no task-specific scripts.
• A feedback-retry loop enabling in-place correction without full trajectory restart.
• Empirical evaluation across 5 VLM evaluators × 3 CUAs, showing consistent success rate improvements.

Results

• Classification accuracy (done/not done vs. human ground truth):
• Claude 3.5 Sonnet: up to 73% (highest, on UI-TARS)
• GPT-4o: up to 69%
• Qwen2-VL-7B: up to 70% (best open-source)
• LLaVA-v1.5-7B: 52–56% (weakest)
• Task success rate improvement after one feedback-retry round:
• Average relative improvement: 27% across all CUA/evaluator pairs
• Proprietary evaluators (GPT-4o, Claude 3.5 Sonnet): up to 61% relative gain
• Weaker baseline agents (Anthropic CU) benefit the most from feedback
• Open-source evaluators (Qwen2-VL-7B) also yield consistent gains

Limitations

• Only one retry round is evaluated; multi-round feedback loops are not studied.
• Binary success metric only — no step-level or partial-completion evaluation.
• Restricted to macOS built-in apps; cross-OS generalization is untested.
• No inter-model agreement or calibration analysis to quantify evaluator reliability variance.
• Tasks exclude scenarios requiring private user data or external account access, limiting real-world coverage.
• Small dataset per app (30 tasks each); difficulty distribution and diversity within apps is not analyzed.

Relevance to Agentic AI / LLM Agents

Self-evaluation and termination detection are fundamental open problems for long-horizon agentic systems — agents that over-run or prematurely stop are both costly and untrustworthy. This work operationalizes VLM-as-judge as a lightweight, script-free, modality-agnostic feedback signal for GUI agents, demonstrating measurable self-correction gains without any training. It directly connects to the broader literature on agentic evaluation (Agent-as-a-Judge, AutoEval) and extends it to unstructured desktop environments where log-based success signals are unavailable, making it relevant to any pipeline deploying CUAs on real operating systems.