Skip to content

Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding

🕒 Published (v1): 2026-03-27 09:34 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; vision-language diffusion models challenge AR dominance for GUI grounding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Autoregressive VLMs dominate GUI grounding, but discrete diffusion VLMs (DVLMs) offer bidirectional attention and parallel decoding yet have never been studied for this task. This work adapts LLaDA-V for GUI grounding and introduces a hybrid masking schedule that decomposes bounding-box prediction into an anchor phase (linear masking) and an extent-refinement phase (full deterministic masking), improving Step Success Rate by up to 6.1 points over the linear-masked baseline and narrowing the gap to autoregressive models from ~25 to under 15 points.

Problem

All prior GUI grounding work uses autoregressive VLMs, which rely on sequential, unidirectional decoding. Discrete DVLMs with bidirectional attention and iterative refinement are a promising alternative, but their applicability to GUI grounding—requiring precise bounding-box coordinate prediction with geometric dependencies among \((x_1, y_1, x_2, y_2)\)—has not been explored. Random token corruption under a standard linear masking schedule fails to reliably expose the model to configurations where extent coordinates are masked while the anchor remains visible, weakening the learned conditional spatial structure.

Method

The base model is LLaDA-V (8B), a discrete DVLM combining a LLaDA language tower, a SigLIP-2 vision encoder, and a two-layer MLP projector. The model is adapted for single-turn action prediction: given screenshot \(I\) and instruction \(N\), it generates an action string \(a = [a_{\text{type}}, B]\) where \(B = (x_1, y_1, x_2, y_2)\) is tokenized and denoised from a fully masked sequence.

The core contribution is a hybrid masking schedule with two complementary phases during training:

  1. Linear masking phase: Standard LLaDA-V schedule with \(p_{\text{mask}} = (1-\varepsilon)t + \varepsilon\), \(t \sim \mathcal{U}(0,1)\). Trains the model to predict action type and anchor coordinates \((x_1, y_1)\) through progressive coarse-to-fine denoising.
  2. Full deterministic masking phase: All response tokens are fully masked; the model is conditioned on image \(I\), instruction \(N\), and anchor \((x_1, y_1)\) to predict extent coordinates \((x_2, y_2)\), explicitly training \(p_\theta(x_2, y_2 \mid a_{\text{type}}, x_1, y_1, I, N)\).

This enforces the conditional geometric structure that random masking produces only rarely. Training data is scaled to 120K samples across web (Mind2Web, WebLinX, OS-Atlas), mobile (OS-Atlas, Rico Widget Caption), and desktop (OS-Atlas) domains. OCR-guided target annotation and cropping around target elements further improve accuracy and latency.

Key Contributions

  • First study applying discrete DVLMs to GUI grounding and single-step action prediction.
  • Hybrid masking schedule that decomposes bounding-box prediction into coarse anchor localization (linear masking) and conditional extent refinement (deterministic masking).
  • Empirical characterization of the accuracy–latency trade-off across diffusion steps, generation length, and block length, identifying 64/64/64 as the optimal setting.
  • Demonstration that multi-domain GUI data scaling reduces latency by ~1.3 s and improves grounding accuracy by an average of 20 points across benchmarks.

Results

  • Mind2Web (7k fine-tune): LLaDA-V 8B with linear masking achieves 80.67% SSR, 99% Action-Type F1; adding OCR annotation and cropping raises SSR to 83.31% with lower latency (4.46 s vs. 4.84 s).
  • Hybrid masking vs. linear masking: Up to +6.1 SSR points on held-out benchmarks.
  • Gap to AR models: Narrowed from ~25 points to under 15 points despite no grounding-specific pretraining.
  • Data scaling (7k → 120k): SSR on ScreenSpot-Web-Text rises from 54.4% to substantially higher; ScreenSpot-Web-Icon from 19.9%; VisualWebArena from 32.4%; latency drops ~1.3 s; converged diffusion steps decrease.
  • Inference parameter ablation: Increasing steps/lengths from 32 to 64 yields +2.5 SSR points but nearly doubles latency; beyond 64 accuracy plateaus while latency continues rising.
  • Baselines include Qwen2.5-VL (3B, 7B) and Phi-3-Vision as AR counterparts; LLaDA-V hybrid masking is competitive but still trails AR models by under 15 points.

Limitations

  • Study restricted to single-turn grounding; multi-step planning and dependent action sequences are explicitly deferred to future work.
  • LLaDA-V was the only publicly available discrete DVLM at time of study, so comparisons across diffusion architectures are absent.
  • Inference latency remains higher than AR models, with the accuracy–latency trade-off requiring empirical tuning per task.
  • Performance is sensitive to image resolution and annotation quality; high-resolution screenshots challenge the SigLIP-2 encoder, requiring domain-specific cropping heuristics.
  • No grounding-specific pretraining was performed; the gap to specialized AR models (e.g., Ui-Tars, OS-Atlas) likely remains substantial.
  • Experiments cover only web-centric benchmarks; generalization to novel desktop/mobile GUIs with out-of-distribution layouts is unverified.

Relevance to Agentic AI / LLM Agents

GUI grounding is a foundational perception-action primitive for digital agents, and this work is directly relevant as the first proof-of-concept that diffusion-based VLMs can replace autoregressive models in this role. The bidirectional attention and iterative refinement properties of discrete DVLMs align well with the spatial reasoning demands of GUI interaction, suggesting a complementary modeling paradigm to the dominant AR approach. The hybrid masking insight—decomposing structured coordinate prediction into coarse-then-fine phases—is a broadly applicable technique for any agent task requiring hierarchical spatial output. For researchers tracking LLM agents, this paper opens a new architecture axis (diffusion vs. autoregression) for the perception module of GUI agents.