Tactile: Giving Computer-Using Agents Hands and Feet¶

🕒 Published (v1): 2026-07-16 00:35 UTC · Source: Arxiv · link

Why this paper was selected

Replaces brittle coordinate-click motor layer for computer-use agents; practical foundation for GUI automation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

TACTILE is an open-source MCP-compatible tool layer that gives computer-using agents structured, semantics-aware "hands and feet" by fusing OS accessibility APIs, OCR, and visual fallback into ranked, action-grounded interface states. It separates target grounding, action execution, and outcome verification into distinct steps rather than collapsing all three into a coordinate click. On macOSWorld-style tasks, it raises Success@100 by ~9 percentage points for Codex and shows consistent gains across four agent backends.

Problem¶

Screenshot-first computer-use agents collapse target grounding, action execution, and outcome verification into a single coordinate prediction, making failures hard to attribute and actions hard to verify. Existing approaches either ignore OS accessibility semantics entirely or treat them as a complete replacement for vision, neither of which works reliably across real desktop applications with incomplete, noisy, or stale accessibility trees.

Method¶

TACTILE implements an accessibility-first operating ladder with three levels: (1) OS accessibility semantics (macOS Accessibility API / Windows UI Automation) for roles, names, states, and native press actions; (2) OCR-grounded coordinates when visible text is the best locator; (3) visual/screenshot fallback for semantically opaque regions (canvases, media editors, remote desktops). These evidence sources are compiled by get_app_state, which normalizes geometry across sources, merges overlapping candidates (e.g., an OCR line overlapping an AX button strengthens that candidate's label), ranks candidates by actionability/stability, and emits a compact model-facing hypothesis set while writing a full provenance trace for replay and failure attribution.

The agent operates through an observe–ground–act–verify loop: the model selects a labeled candidate (not an anonymous coordinate), the runtime chooses the safest executable primitive (native semantic action > coordinate click), and post-action state is re-observed to produce explicit verification evidence (or record its absence). The tool surface is exposed as an MCP server (list_apps, get_app_state, perform_secondary_action, click, type_text, press_key, scroll, drag), making it agent-agnostic.

Key Contributions¶

Formalization of computer use as an action-grounded interface problem where targetability, actionability, verifiability, and auditability are first-class runtime properties.
The accessibility-first operating ladder that composes AX semantics, OCR coordinates, and visual fallback in a principled priority order without reducing to AX-only.
TACTILE as an open MCP tool layer with normalized coordinate contracts, compact candidate generation, and full trace provenance for replay and failure attribution.
Empirical evaluation on macOSWorld-style tasks across four agent backends (Codex, Claude Code, OpenCode, Goose), showing consistent Success@100 improvements.
The reframing of human accessibility infrastructure as shared agent infrastructure: improving WCAG/AX compliance directly improves agent operability.

Results¶

Codex (full graded set): Success@100 41.1% → 50.0% (+8.94 pp overall); AX-adapted tasks 45.2% → 55.3% (+10.04 pp); Limited-AX tasks 27.8% → 33.3% (+5.55 pp).
96-task cross-agent subset (skill-optional upper bound vs. no-skill baseline):
Codex: 38.5% → 50.0% overall; AX-adapted 46.0% → 59.5%
Claude Code: 33.3% → 43.75% overall; AX-adapted 39.2% → 51.4%
OpenCode: 33.3% → 40.6% overall; AX-adapted 39.2% → 47.3%
Goose: 41.7% → 43.75% overall; AX-adapted 48.6% → 51.4%
Limited-AX gains are smaller across all agents; Goose shows no gain on Limited-AX tasks, consistent with TACTILE deriving most value from semantically exposed interfaces.
Trace-based case study (Zoom scheduling): 11 state snapshots, up to 468 accessibility candidates in a single snapshot, traversal times 0.24–0.85 s.

Limitations¶

Incomplete/noisy AX metadata: Many controls expose roles without names; some actions are listed but produce no reliable state changes; stale trees are common.
macOS-centric implementation: Cross-platform support (Windows UI Automation) is early; sandboxing, webview, and browser permission models vary significantly across platforms.
Large/dynamic UI trees: Candidate compaction can filter out low-ranked elements that later become task-critical; no query-driven or task-conditioned re-ranking yet.
OCR and visual fallback failures: Low-contrast text, icon-only controls, animations, and unusual fonts degrade OCR; visual fallback reintroduces coordinate-level ambiguity.
Verification gap: Some applications provide no reliable post-action feedback; TACTILE records the absence of verification evidence but cannot resolve it.
Cross-agent score is an upper bound: The reported "With Tactile" cross-agent metric is the best per-task result across conditions, not a learned routing policy—actual deployable gains may be lower.

Relevance to Agentic AI / LLM Agents¶

TACTILE directly addresses one of the core infrastructure gaps in computer-use agents: the absence of a reusable, semantics-aware execution substrate between the LLM planner and raw OS primitives. By exposing desktop state as structured, verifiable, auditable objects rather than pixel coordinates, it decouples planning from execution and makes failures attributable—both critical properties for reliable agentic loops. Its MCP compatibility means any LLM agent client can adopt it without model changes, positioning it as shared infrastructure analogous to what browser DevTools or DOM APIs provide for web agents. The alignment between human accessibility requirements and agent operability also points toward a practical incentive structure for improving agent-ready software ecosystems.