CoAct-1: Computer-using Multi-agent System with Coding Actions

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; computer-use multi-agent system combining planning with coding actions

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoAct-1 is a three-agent system (Orchestrator, Programmer, GUI Operator) that augments GUI-based computer control with direct Python/Bash code execution, enabling agents to bypass brittle click sequences for backend tasks. It achieves state-of-the-art 60.8% on OSWorld and 52.5% on WindowsAgentArena while cutting average task steps from ~15 to 10.15.

Problem

Pure-GUI agents suffer from cumulative error propagation over long action sequences: a single mis-click can derail a task, and even planner-augmented GUI systems remain bottlenecked by low-level GUI execution for operations (file management, spreadsheet manipulation, data processing) that are inherently better suited to programmatic control. The action space \(\mathcal{A} = \mathcal{A}_{\text{GUI}}\) is both inefficient and brittle for long-horizon tasks.

Method

CoAct-1 extends the action space to \(\mathcal{A} = \mathcal{A}_{\text{GUI}} \cup \mathcal{A}_{\text{Code}}\) and implements a hierarchical policy \(\pi\) through three specialized agents:

• Orchestrator (\(\pi_{\text{Orch}}\), powered by OpenAI o3): receives the user goal \(G\) and history \(H_t = (o_1, a_1, \ldots, o_t)\), decomposes tasks, and dynamically delegates each subtask to either the Programmer or GUI Operator.
• Programmer (\(\pi_{\text{Code}}\), powered by o4-mini): receives subtasks requiring backend operations and enters a multi-round code-interpreter loop generating Python/Bash scripts, observing execution feedback, and refining until the subtask is solved.
• GUI Operator (\(\pi_{\text{GUI}}\), powered by OpenAI computer-use-preview): handles visual/frontend tasks via the standard perception-action loop (screenshot → action → new screenshot).

A dedicated summarizer model (o4-mini) condenses each executor's full interaction history into a compact summary returned to the Orchestrator, enabling isolated working memories per executor (cleared after each subtask) while the Orchestrator maintains the global long-horizon context \(H_t\). The system is implemented using the AG2 multi-agent framework with a remote RESTful code interpreter.

Key Contributions

• A hybrid action space formalizing \(\mathcal{A} = \mathcal{A}_{\text{GUI}} \cup \mathcal{A}_{\text{Code}}\) for computer-use agents, with dynamic routing at the Orchestrator level.
• A three-agent architecture with hierarchically isolated memories — long-term (Orchestrator) vs. short-term working memory (executors), cleared on subtask completion.
• State-of-the-art results on OSWorld and WindowsAgentArena, demonstrating cross-OS generalization of the hybrid paradigm.
• Demonstrated efficiency gains: coding actions replace long GUI sequences, reducing average solved-task steps from ~15 (GTA-1) to 10.15.

Results

• OSWorld (369 tasks, 150-step budget): 60.76% overall success rate vs. Agent S2.5 w/ o3 (55.98%), GTA-1 w/ o3 (53.07%), Jedi-7B w/ o3 (50.98%).
• At the 100-step mark CoAct-1 already achieves 59.93%, exceeding all baselines' final scores.
• Category highlights on OSWorld: OS tasks 75.0%, Professional (VSCode/GIMP) 71.82%, Multiple Apps 47.87%, Office 64.80%.
• WindowsAgentArena (154 tasks, 100-step budget): 52.5% vs. Agent S2 (29.8%) and NAVI (19.5%); Windows System 83.3%, Windows Utils 77.7%.
• Efficiency: average steps per solved task = 10.15 (CoAct-1) vs. 15.22 (GTA-1), 14.90 (UI-TARS-7B), 15.00 (OpenAI CUA 4o).
• Clear step-budget scaling on WindowsAgentArena: 21.4% @ 15 steps → 43.5% @ 50 steps → 52.5% @ 100 steps.

Limitations

• Relies on frontier closed-source models (o3, o4-mini, computer-use-preview), making cost, reproducibility, and open-source replication difficult.
• Maximum theoretical step ceiling is 375, and failure rate is positively correlated with total steps required — inherently hard tasks still fail.
• Evaluation is restricted to two benchmarks (Linux/Windows); generalization to web-only or mobile environments is undemonstrated.
• The Programmer requires file paths and system context to be correctly inferred and passed by the Orchestrator — grounding errors at the orchestration level can cascade to code failures.
• No ablation isolating the contribution of each backbone model choice from the architectural design.

Relevance to Agentic AI / LLM Agents

CoAct-1 directly advances the design space of tool-augmented LLM agents by showing that expanding the action space beyond a fixed GUI modality — enabling agents to self-select code execution as an action primitive — yields large gains in both reliability and efficiency. The hierarchical orchestration with isolated, ephemeral executor memories is a concrete architectural pattern for managing context across long-horizon multi-step tasks, directly relevant to research on memory management and agent modularity. The result that programmatic actions most benefit structured data and file-system tasks informs where hybrid GUI+code approaches should be prioritized over pure end-to-end GUI agents. For researchers tracking agentic AI, this paper is an existence proof that the choice of action space, not just the underlying model, is a first-class design variable in building capable computer-use agents.