Skip to content

Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding

🕒 Published (v1): 2026-03-27 09:34 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; diffusion VLMs for GUI grounding challenge autoregressive model dominance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper is the first to explore discrete diffusion vision-language models (DVLMs) as an alternative to autoregressive (AR) models for GUI grounding — the task of localizing interface elements from natural-language instructions. The authors adapt LLaDA-V for single-turn action and bounding-box prediction and propose a hybrid masking schedule that captures the geometric hierarchy of bounding-box coordinates, improving Step Success Rate (SSR) by up to 6.1 points over the standard linear-masked baseline.

Problem

AR VLMs dominate GUI grounding but are architecturally constrained by sequential decoding and unidirectional attention. Discrete DVLMs offer bidirectional attention, parallel token generation, and iterative refinement, yet their applicability to GUI grounding — which requires learning structured geometric dependencies among bounding-box coordinates \(B = (x_1, y_1, x_2, y_2)\) — was entirely unexplored. Specifically, the standard linear masking schedule of LLaDA-V corrupts tokens randomly, which rarely enforces the conditional configuration needed to learn \(p_\theta(x_2, y_2 \mid a_\text{type}, x_1, y_1, I, N)\).

Method

The authors adapt LLaDA-V (8B; architecture: LLaDA language tower + SigLIP-2 vision tower + MLP projector) for single-turn GUI grounding by formulating the task as text generation from a screenshot and natural-language instruction, predicting an action string \(a = [a_\text{type}, B]\) where \(a_\text{type} \in \{\texttt{lclick}, \texttt{hover}, \texttt{type\_in}\}\) and coordinates are normalized to \([0, 1000]\).

The core contribution is a hybrid masking schedule with two complementary training phases:

  1. Linear Masking Phase: Retains the standard LLaDA-V forward process with \(p_\text{mask} = (1-\varepsilon)t + \varepsilon\), \(t \sim \mathcal{U}(0,1)\). Trains the model to predict action type and anchor coordinates \((x_1, y_1)\) under variable corruption.

  2. Full Deterministic-Masking Phase: All response tokens are fully masked during the forward process. The model is trained to predict extent coordinates \((x_2, y_2)\) conditioned on the image, instruction, and anchor \((x_1, y_1)\), enforcing the conditional relationship that linear masking rarely produces.

Together, the two phases implement a coarse-to-fine spatial learning regime. Training data is scaled to 120K samples spanning web (Mind2Web, WebLinX, OS-Atlas), mobile (OS-Atlas, Rico Widget Caption), and desktop (OS-Atlas) domains. Inference uses 64 diffusion steps, generation length 64, and block length 64 as the empirically optimal configuration.

Key Contributions

  • First study applying discrete DVLMs to GUI grounding and single-step action prediction.
  • Hybrid masking schedule that decomposes bounding-box prediction into an anchor phase (linear masking) and an extent phase (deterministic full masking), capturing geometric hierarchy.
  • Empirical characterization of the accuracy–latency trade-off as a function of diffusion steps, generation length, and block length.
  • Demonstration that multi-domain data scaling (7K→120K) reduces inference latency by ~1.3 s and improves grounding accuracy by an average of ~20 points across benchmarks.
  • Evidence that OCR-guided annotations and resolution-aware cropping are critical for GUI grounding performance with discrete DVLMs.

Results

  • Mind2Web (7K fine-tune): LLaDA-V with linear masking achieves 80.67% SSR, 99% action-type F1 at 4.84 s latency (64 steps); cropping + OCR annotation raises this to 83.31% SSR at 4.46 s.
  • Hybrid masking improvement: Up to +6.1 points SSR over the linear-masked LLaDA-V variant across benchmarks.
  • Gap to AR models: Hybrid masking narrows the gap from ~25 points to under 15 points versus AR baselines (Qwen2.5-VL 3B/7B, Phi-3-Vision), without grounding-specific pretraining.
  • Data scaling (7K→120K): SSR on ScreenSpot-Web-Text 54.4%→ higher; ScreenSpot-Web-Icon 19.9%→ higher; VisualWebArena 32.4%→ higher (exact scaled numbers not fully reported in the provided text); latency reduced ~1.3 s; converged diffusion steps reduced.
  • Zero-shot LLaDA-V (no GUI fine-tuning) performs near zero on all benchmarks, confirming task-specific fine-tuning is necessary.
  • Inference parameter sensitivity: raising all parameters from 32→64 yields +2.5 points SSR but nearly doubles latency; beyond 64, accuracy plateaus at ~80%.

Limitations

  • Performance still lags AR models by ~15 points even with hybrid masking, partly attributable to limited GUI-specific pretraining.
  • Only single-turn, atomic grounding is studied; multi-step planning and dependent action sequences are out of scope.
  • Inference latency is substantially higher than AR models and increases with diffusion steps.
  • LLaDA-V was the only publicly available discrete DVLM at time of study, limiting architectural comparison.
  • Sensitive to input resolution and annotation quality (OCR granularity); high-resolution screenshots challenge the SigLIP-2 encoder.
  • Block length ≥ generation length increases latency with negligible accuracy gains, constraining the practical benefit of semi-autoregressive generation.

Relevance to Vision-Language Models

This work challenges the AR paradigm that has dominated GUI-grounding VLMs (CogAgent, SeeClick, UGround, OS-Atlas) by demonstrating that discrete diffusion VLMs can viably perform spatially structured, action-grounded prediction. The bidirectional attention inherent to masked diffusion is particularly pertinent for VLMs that must jointly resolve semantic intent and spatial coordinates — a capability that unidirectional AR decoding handles less naturally. The hybrid masking strategy is a transferable technique for any discrete DVLM task requiring structured, hierarchically dependent output sequences. For VLM researchers, this opens a new design axis: masking schedules as an inductive bias for geometric reasoning, not just language fluency.