Skip to content

Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents

🕒 Published (v1): 2026-06-30 07:44 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; inference-time self-improvement for computer-use; Schmidt and Yeung-Levy groups

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current computer-use agent self-improvement pipelines fine-tune only on successful trajectories, discarding the majority of experience. This work introduces a complementary failure-case loop that uses an LLM to diagnose failed trajectories, categorize failure modes, and generate code patches applied at inference time—no retraining required. Applied to OpenCUA-72B on OSWorld, it yields a +6.6 absolute point gain (42.3% → 48.9%) with modest overhead.

Problem

Success-driven self-training loops (agent → environment → successful trajectory → SFT → improved agent) discard all failed trajectories despite the engineering cost of verifiable environments. Failed trajectories contain structured diagnostic signals about model weaknesses—grounding errors, capability gaps, knowledge holes, and repetitive loops—that are entirely wasted in current pipelines.

Method

The failure-case loop operates iteratively in four rounds:

  1. Failure collection: The agent rolls out tasks in a verifiable environment; failed trajectories (instruction + action history + reasoning trace) are retained.
  2. LLM-guided diagnosis: An LLM acts as a meta-controller, reading each failed trajectory and categorizing the dominant failure mode from a taxonomy it develops.
  3. Solution synthesis: The LLM proposes an inference-time fix and generates a code patch implementing it.
  4. Lightweight human verification: A human minimally reviews and integrates the patch into the agent's inference pipeline.

Four strategies are introduced over successive rounds: - Visual Search (grounding errors): after a spatially grounded action (click/moveto/dragto), crop a fixed region around the target, overlay a red marker, and prompt the agent to self-verify and revise coordinates. - Terminal Execution (competency gaps): unlock Ctrl+Alt+T terminal access so the agent can substitute precise CLI commands for brittle GUI interactions. - Knowledge Support (knowledge deficiencies): inject a software-manual retriever (curated hotkey/procedure references) and a search-engine interface (querying an external LLM, GPT-5-mini) when domain-specific knowledge is missing. - Repetition Warnings (redundant loops): a sliding-window monitor tracks thought, action, and screen-state traces; if thought/action/screen-state repetition exceeds a threshold, a recovery mode forces the agent to reformulate or switch strategy.

All patches accumulate across rounds; no weight updates occur.

Key Contributions

  • Identification of the fundamental inefficiency: failed trajectories are discarded even though they carry structured diagnostic signal.
  • A concrete failure-case loop framework: LLM-as-meta-controller → failure taxonomy → code patch → lightweight human gate → inference-time behavioral upgrade.
  • Four empirically grounded failure categories (grounding errors, competency gaps, knowledge deficiencies, redundant loops) and four matching inference-time mitigations.
  • First empirical demonstration that a failure-driven loop produces meaningful gains on a competitive computer-use benchmark without any additional training.

Results

  • OSWorld full set (100 steps): OpenCUA-72B improves from 42.3% → 48.9% (+6.6 pp absolute).
  • OSWorld small set (30 steps), ablation:
  • Baseline: 41.67%
    • Visual Search: 47.22% (+5.55 pp)
    • Repetition Detection: 44.40% (+2.73 pp)
    • Terminal Execution: 47.19% (+5.52 pp)
    • Knowledge Support: 44.44% (+2.77 pp)
  • Full method (all combined): 52.74% (+11.07 pp)
  • Initial failure-mode distribution: action loops 43%, coordinate precision 24%, UI recognition 19%, multi-step planning 15%.
  • After applying all strategies, action loops drop substantially and multi-step planning becomes the dominant remaining failure mode (28%), reflecting a shift toward higher-level cognitive bottlenecks.
  • Consistent improvements across multiple GUI benchmarks and across agent families/scales; stronger models benefit the most.

Limitations

  • Human verification of LLM-generated patches remains a required step, preventing fully autonomous operation.
  • The taxonomy of four failure modes was derived empirically from OSWorld; generalization to qualitatively different environments is asserted but not exhaustively demonstrated.
  • The paper is truncated before the full generalization results are shown; claims about cross-benchmark and cross-agent consistency are partly from the abstract.
  • CAPTCHA tasks (~11-12% of failures post-improvement) represent a ceiling the framework cannot address.
  • Inference overhead, while described as "modest," is not quantified in the provided text.
  • The LLM meta-controller cost (API calls for failure analysis) is not accounted for in the reported overhead.

Relevance to Agentic AI / LLM Agents

This paper directly addresses a systemic weakness in agentic self-improvement pipelines: the data efficiency of learning from failure rather than only from success. The failure-case loop is a practical, training-free complement to SFT-based loops and is applicable to any agent operating in a verifiable environment. The four-category taxonomy (grounding, competency, knowledge, loops) provides a reusable diagnostic framework for understanding GUI-agent failure patterns. The result that stronger models benefit the most suggests compounding value as base models improve, making this approach increasingly relevant as frontier computer-use agents scale.