Skip to content

VisCritic: Visual State Comparison as Process Reward for GUI Agents

🕒 Published (v1): 2026-06-23 12:57 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; visual-state process reward for GUI agents — directly useful for agent eval harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VisCritic is a plug-and-play visual process reward model for GUI agents that verifies action outcomes by comparing pre- and post-action screenshots in learned semantic feature space, rather than relying on text-based reasoning. A Siamese ViT extracts patch-level difference features, which a multi-task critic head uses to predict action success, task progress, and error type. Evaluated across five benchmarks, it consistently outperforms text-based process reward baselines and improves diverse base agents at inference time without retraining them.

Problem

Existing GUI process reward models (PRMs) verify agent actions through textual reasoning, tool calls, or structured checklists. This creates a modality gap: GUI action outcomes manifest as pixel-level visual state changes (button highlights, dialogs appearing, page transitions) that are fundamentally difficult to describe or verify in text. Without reliable step-level visual verification, long-horizon GUI agents accumulate errors silently, causing cascading failures.

Method

VisCritic defines a critic function \(C : (s_t, a_t, s_{t+1}, l) \to (\hat{y}_\text{suc}, \hat{y}_\text{prog}, \hat{y}_\text{err})\) mapping screenshot pairs to success probability, progress score, and error category.

Visual Difference Encoder (VDE): A shared-weight ViT (Siamese architecture) encodes \(s_t\) and \(s_{t+1}\) into patch features \(F_t, F_{t+1} \in \mathbb{R}^{P \times d}\). Patch-level differences \(\Delta F = F_{t+1} - F_t\) and a change magnitude map \(M_i = \|\Delta F_i\|_2\) are computed. Change region attention \(\alpha_i = \exp(M_i/\beta) / \sum_j \exp(M_j/\beta)\) with learnable temperature \(\beta\) weights the attended difference vector \(\mathbf{v}_\Delta = \sum_i \alpha_i \cdot \Delta F_i\).

Action-Aware Critic Head: Cross-attention fuses \(\Delta F\) (queries) with text-encoded action \(\mathbf{h}_a\) and instruction \(\mathbf{h}_l\) (keys/values), combined with spatial self-attention over \(\Delta F\) via change-magnitude-weighted pooling. Three MLP heads predict: \(\hat{y}_\text{suc} = \sigma(\cdot)\), \(\hat{y}_\text{prog} = \tanh(\cdot)\), \(\hat{y}_\text{err} = \text{softmax}(\cdot)\).

Training: Annotation-free data construction from existing trajectories generates ~202K samples (positive steps from successful trajectories; four negative types: action mismatch, state mismatch, no-op, failed trajectory). Phase 1 contrastive pre-training uses InfoNCE loss with an auxiliary KL-divergence localization loss. Phase 2 multi-task fine-tuning uses \(\mathcal{L} = \lambda_1 \mathcal{L}_\text{BCE} + \lambda_2 \mathcal{L}_\text{MSE} + \lambda_3 \mathcal{L}_\text{CE}\).

Integration: Primary mode is post-execution verification; optional Best-of-N pre-selection is supported when a state predictor is available.

Key Contributions

  • Identification of the modality gap between textual verification and visual GUI state changes; visual state comparison proposed as a more natural PRM signal.
  • Siamese VDE operating in learned semantic feature space (not raw pixels) for meaningful change representation.
  • Multi-task Action-Aware Critic Head jointly predicting success, progress, and error type with cross-attention fusion of visual differences and textual context.
  • Annotation-free critic-training data pipeline with 81.4% error-label agreement against 500 manually annotated cases.
  • Plug-and-play integration demonstrated across four diverse base agents (SeeClick, ShowUI, CogAgent, Qwen2.5-VL) on five benchmarks.

Results

  • Overall: VisCritic improves interactive average over all four base agents: SeeClick +3.9 pts (16.8→20.7), ShowUI +4.6 pts (22.8→27.4), CogAgent +2.9 pts (20.7→23.6), Qwen2.5-VL +5.4 pts (27.2→32.6).
  • vs. text-based PRMs: Outperforms GUI-PRA, BacktrackAgent, GuidNav, and GUI-Critic-R1 in the majority of benchmark–agent settings under a unified re-implementation protocol.
  • Verification quality (AndroidWorld, Qwen2.5-VL): VisCritic achieves 85.2% F1 vs. GUI-PRA 70.4% and BacktrackAgent 69.5%.
  • Controlled modality comparison (Tab. 3): VisCritic 84.6% overall F1 vs. Text-Critic 66.2% and Pixel-Critic 70.5%; most pronounced gap on visual-only changes (84.3% vs. 58.3%) and no-op detection (84.2% vs. 62.7%).
  • Ablations (ShowUI on AndroidWorld): Removing post-execution verification −3.4 pts; replacing VDE with pixel diff −3.6 pts; removing contrastive pre-training −2.8 pts; removing change region attention −1.4 pts.
  • Zero-shot generalization: OSWorld and WebArena are held out from training; gains are demonstrated cross-benchmark and cross-platform.
  • One exception: Qwen2.5-VL on OSWorld, where a text-based critic marginally outperforms VisCritic, attributed to text-heavy desktop environment characteristics.

Limitations

  • AndroidWorld improvements do not reach conventional statistical significance (\(p \approx 0.08\)) due to the small 34-task held-out split; p-values are uncorrected for multiple comparisons across benchmarks.
  • VisCritic (~13B: InternViT-6B + Qwen2.5-7B) is larger than some baselines, though controlled experiments with SigLIP-400M (~7.4B) still show advantage.
  • Best-of-N pre-execution mode requires an external visual state predictor (e.g., MobileDreamer), limiting its applicability.
  • Recovery policy design (retry/backtrack logic) is outside the main experimental scope; gains reflect a fixed integration hook rather than optimized recovery.
  • AITW results are an offline logged-screenshot diagnostic, not live inference-time gains.
  • Text-heavy desktop environments (OSWorld) may favor textual verification, suggesting VisCritic's advantage is environment-dependent.

Relevance to Vision-Language Models

VisCritic directly addresses a core limitation of VLM-based GUI agents: the reliance on language-mediated reasoning to verify outcomes that are inherently visual. By using a Siamese ViT to compare VLM agent screenshots in feature space, it demonstrates that the verification signal itself should be modality-matched to the domain. This connects to broader VLM research on grounding, self-correction, and process supervision—VisCritic is complementary to textual PRMs and could be combined with them for multimodal process reward. The annotation-free training pipeline is particularly relevant: it shows how existing VLM agent trajectory datasets can be repurposed to train lightweight verifiers without human labeling, a scalable direction for improving VLM agent reliability.