GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GUI-Actor replaces text-based coordinate generation for GUI visual grounding with an attention-based action head that learns to align a dedicated <ACTOR> token with relevant visual patch tokens, producing coordinate-free region proposals. A lightweight grounding verifier then selects the most plausible candidate from the attention map. GUI-Actor-7B achieves 40.7/44.6 on ScreenSpot-Pro (with Qwen2-VL/Qwen2.5-VL backbones), outperforming UI-TARS-72B (38.1) with far fewer parameters.

Problem¶

Existing GUI grounding methods treat the task as text-based coordinate generation (outputting x/y values as language tokens), which suffers from three fundamental flaws: (1) weak spatial-semantic alignment because no explicit spatial supervision connects visual features to numeric outputs; (2) ambiguous supervision since single-point annotations penalize all valid click variations within a UI element; and (3) a granularity mismatch between the continuous, high-resolution coordinate space and the coarse patch-level features of Vision Transformers.

Method¶

GUI-Actor introduces three components on top of a frozen or fine-tuned VLM backbone (Qwen2-VL):

<ACTOR> token as contextual anchor: the model output replaces coordinate tokens with three special tokens (<ACTOR_START>, <ACTOR>, <ACTOR_END>); the final-layer hidden state of <ACTOR> serves as the grounding query.
Attention-based action head (~20M/100M params for 2B/7B): a self-attention layer aggregates spatially neighboring visual patch features, then two separate MLPs project the <ACTOR> hidden state and patch features into a shared space, and a dot-product attention score over all M patch tokens produces a spatial activation map identifying target regions—in a single forward pass, yielding multiple diverse candidate regions.
Spatial-aware multi-patch supervision: all ViT patches partially or fully overlapping the ground-truth bounding box are labeled positive; training combines next-token prediction loss with a KL-style action attention loss pushing the attention distribution toward the bounding-box-covering patches.
Grounding verifier (fine-tuned from UI-TARS-2B-SFT): takes a screenshot with a hollow red circle marker at a candidate location and predicts True/False; top-K candidates are evaluated in descending attention-weight order and the first exceeding a confidence threshold γ is selected.

Key Contributions¶

A coordinate-free GUI grounding architecture replacing numeric coordinate generation with patch-level attention from a dedicated <ACTOR> token.
Multi-patch bounding-box supervision that tolerates spatial ambiguity and provides dense, spatially structured training signals.
A lightweight grounding verifier that reuses diverse single-pass candidates to refine the final action location without additional inference overhead on the main model.
A lite-training variant (GUI-Actor-LiteTrain) that freezes the backbone VLM and trains only ~19M/103M parameters, preserving general-purpose capabilities while enabling GUI grounding.
Demonstrated improved sample efficiency: GUI-Actor reaches peak accuracy at ~60% of training data vs. 80–90% for coordinate-based baselines.

Results¶

ScreenSpot-Pro (high-resolution, OOD): GUI-Actor-7B (Qwen2-VL) 40.7 vs. UI-TARS-7B 35.7, UI-TARS-72B 38.1; +Verifier → 44.2. With Qwen2.5-VL: GUI-Actor-7B 44.6, +Verifier 47.7 vs. Jedi-7B 39.5.
ScreenSpot: GUI-Actor-7B 88.3 vs. UI-TARS-7B 89.5 (comparable), Aguvis-7B 84.4; +Verifier → 89.7.
ScreenSpot-v2: GUI-Actor-7B 89.5 vs. UI-TARS-7B 91.6; +Verifier → 90.9.
GUI-Actor-2B vs. 7B competitors: GUI-Actor-2B scores 36.7/86.5/88.6 on ScreenSpot-Pro/ScreenSpot/ScreenSpot-v2, surpassing many 7B baselines (e.g., UGround-V1-7B 31.1/86.3/87.6).
LiteTrain + Verifier (103M params updated): 35.8 on ScreenSpot-Pro, 81.3 on ScreenSpot, 83.8 on ScreenSpot-v2—competitive with fully fine-tuned coordinate models.
Hit@3 vs. Hit@1: GUI-Actor-7B improves from 40.7 to 45.4 on ScreenSpot-Pro (Hit@3), whereas coordinate-based baselines show negligible gains because repeated sampling produces nearly identical outputs.

Limitations¶

Performance on ScreenSpot and ScreenSpot-v2 remains marginally below UI-TARS-7B, which uses proprietary data, continual pre-training, annealing, and DPO.
The grounding verifier adds inference latency proportional to K candidates evaluated sequentially; the paper does not quantify this overhead.
The action head's resolution is bounded by the ViT backbone's native patch grid, which may limit sub-patch-level precision for very small UI targets.
Training data recipe (~1M screenshots) is not fully public; full reproducibility depends on access to OS-Atlas and other curated datasets.
Evaluation is restricted to element-accuracy grounding benchmarks; end-to-end task success on agents (OSWorld, AndroidWorld) is not reported.

Relevance to Vision-Language Models¶

GUI-Actor demonstrates a principled alternative to coordinate-token generation within VLMs—instead of forcing spatial reasoning through the language modeling head, it attaches a lightweight spatial attention module that exploits the ViT's native patch representations, a technique transferable to any VLM. The lite-training result (freezing the VLM backbone while training only ~100M parameters) is particularly relevant to the VLM community as it shows GUI grounding can be modularly injected without catastrophic forgetting of general vision-language capabilities. The multi-patch bounding-box supervision strategy addresses a broader supervision-ambiguity problem that applies beyond GUIs to other dense grounding tasks (referring expression comprehension, visual question answering with spatial answers). The verifier-as-refinement pattern—generating diverse hypotheses in one forward pass and selecting via a binary scorer—is an increasingly common paradigm in VLM-based agentic systems.