Skip to content

OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use

🕒 Published (v1): 2025-08-06 14:33 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A comprehensive survey of OS Agents — MLLM-based agents that automate tasks on computing devices (desktop, mobile, web) by operating through OS-provided GUIs. It taxonomizes the field across three axes: fundamentals (environment/observation/action), construction (foundation models + agent frameworks), and evaluation (protocols and benchmarks). The paper consolidates the state of the field as of late 2024 and identifies key open challenges in safety, privacy, and personalization.

Problem

No unified survey existed to consolidate the rapidly expanding research on (M)LLM-based OS Agents across diverse platforms, architectures, training strategies, and evaluation protocols. Prior work was fragmented, making it difficult to understand shared components, methodological choices, or open problems across the desktop, mobile, and web sub-communities.

Method

Structured literature survey organized around three tiers: 1. Fundamentals — defines the three key components (Environment, Observation Space, Action Space) and three core capabilities (Understanding, Planning, Grounding) that every OS Agent must possess. 2. Construction — reviews (a) domain-specific foundation model design: architectures (existing LLMs, existing MLLMs, concatenated MLLMs, modified MLLMs with high-res encoders like CogAgent's 1120×1120 EVA-CLIP-L), training strategies (pre-training on GUI grounding/OCR/screen-understanding data, SFT, RL); and (b) agent framework components: perception, planning (e.g., ReAct, CoAT), memory, and action modules. 3. Evaluation — catalogs evaluation principles, metrics, platforms, benchmark settings, and task taxonomies used across the literature.

Key Contributions

  • Unified taxonomy of OS Agents covering environment (desktop/mobile/web), observation modalities (screenshots, HTML, A11y trees), and action types (input, navigation, extended/tool-use).
  • Structured comparison of 30 foundation models (Table 1) across architecture type and training strategy (PT/SFT/RL).
  • Review of agent framework design patterns: perception preprocessing, planning strategies, memory augmentation, and action grounding.
  • Survey of evaluation protocols and benchmarks, including discussion of evaluation platforms and task categories.
  • Identification of open challenges: adversarial attacks on GUI agents, defense mechanisms, privacy risks from agentic OS access, personalization, and self-evolution.
  • Maintained open-source GitHub repository as a live resource.

Results

This is a survey paper; no new experimental results are reported. Cited systems include: - OS-Atlas: foundational GUI action model achieving strong out-of-distribution performance via cross-platform grounding data synthesis. - OS-Copilot: generalist agent framework demonstrating robust generalization and self-improvement across diverse tasks with minimal supervision. - CogAgent: modified MLLM with dual-resolution encoders (224×224 + 1120×1120) showing improved GUI understanding on high-resolution screenshots. - MobileVLM: data collection algorithm yielding >3 million real GUI interaction samples.

No head-to-head aggregate benchmark numbers are reported in the surveyed text.

Limitations

  • No new empirical results; claims about relative system performance are inherited from cited papers without independent re-evaluation.
  • Coverage is bounded by literature available through ~late 2024; rapidly evolving field means the taxonomy may be quickly outdated.
  • The survey text provided is truncated, so the evaluation benchmark section (§4.2) and the full challenges section (§5) are only partially visible.
  • Benchmark comparisons across systems are not normalized — different works use different splits, environments, and metrics, making direct comparison difficult.

Relevance to Harnesses / Meta-Harnesses

OS Agents are themselves a class of execution harnesses: they wrap foundation models with perception pipelines, planning loops, memory modules, and action executors to drive multi-step task completion across arbitrary OS environments — the same orchestration pattern that defines a meta-harness. The survey's breakdown of agent framework components (perception → planning → memory → action) maps directly onto the stages a meta-harness must coordinate. The discussion of non-tuning agent frameworks that compose pre-trained models via reasoning strategies and memory augmentation is particularly relevant: it describes how to build capable harnesses without retraining the underlying model. Challenges around dynamic replanning, cross-platform generalization, and self-evolution in §5 reflect open engineering problems that any serious harness designer must address.