Skip to content

UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction as Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multi-perspective instruction reasoning improves GUI grounding for agent navigation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UI-Ins proposes an Instruction-as-Reasoning paradigm for GUI grounding, treating natural-language instructions not as static inputs but as dynamic analytical pathways (appearance, functionality, location, intent) that the model actively selects and composes during inference. A two-stage SFT+GRPO training framework instills multi-perspective reasoning and then learns to select the optimal perspective per scenario. UI-Ins-32B achieves SOTA on five GUI grounding benchmarks, and UI-Ins-7B as a grounding executor under a GPT-5 planner reaches 74.1% task success on AndroidWorld.

Problem

GUI grounding models are trained on a single, fixed instruction style, ignoring two under-studied variables: (1) instruction diversity — humans fluidly switch among analytical perspectives to describe targets, and exploiting this diversity unlocks up to 76% relative performance gain even without retraining; (2) instruction quality — a manual audit of 1,909 samples from OS-Atlas, AMEX, and Widget Captioning finds 23.3% of annotations are flawed (ambiguous or mismatched), directly degrading model training.

Method

Data pipeline. Ground-truth bounding boxes are refined via OmniParser V2 with IoU-based filtering to remove noisy anchors. GPT-4.1 then generates four instruction variants per sample — Appearance, Functionality, Location, Intent — each verified by GPT-4.1 to ensure unambiguous one-to-one element mapping.

Instruction-as-Reasoning (two stages).

  1. SFT stage. Given screenshot \(S\) and instruction \(I\), the model is trained to first emit a reasoning trace \(R_{gt}\) (a rewritten instruction randomly sampled from one augmented perspective) and then a coordinate \(p_{gt}\). Objective: \(\max_\theta \sum_{(S,I,Y_{gt})\in\mathcal{D}} \log P(Y_{gt}|S,I;\theta)\) where \(Y_{gt} = R_{gt} \oplus p_{gt}\).

  2. RL stage (GRPO). The SFT prompt is replaced with an open-ended "think before answering" instruction — no explicit perspective list — to encourage unconstrained exploration. Reward is binary point-in-box. Group-normalized advantages \(\hat{A}_{i,t}\) are computed across \(G\) rollouts and the policy is updated via: \(\mathcal{L} = -\frac{1}{G}\sum_{i=1}^{G}\frac{\pi(o_i|I,S)}{\pi_{\text{old}}(o_i|I,S)}\cdot\hat{A}_{i,t}\).

Backbone: Qwen2.5-VL-7B and Qwen2.5-VL-32B.

Key Contributions

  • Systematic quantitative analysis showing 23.3% instruction flaw rate in major GUI grounding datasets and a 76% relative performance ceiling from optimal perspective selection.
  • Instruction-as-Reasoning paradigm: instructions as dynamic reasoning pathways rather than static strings.
  • SFT+GRPO training framework that first teaches diverse reasoning then optimizes perspective selection; shown to mitigate policy collapse that plagues coordinate-only SFT baselines.
  • UI-Ins-7B and UI-Ins-32B achieving SOTA across five benchmarks (UI-I2E-Bench, ScreenSpot-Pro, ScreenSpot-V2, MMBench-GUI L2, ShowDown).
  • Emergent reasoning: post-training, the model composes novel perspectives not seen during SFT, and combines multiple perspectives autonomously.

Results

  • UI-I2E-Bench: UI-Ins-32B scores 87.3% avg (92.9 explicit / 83.9 implicit); next best GTA1-32B at 83.5%.
  • MMBench-GUI L2: UI-Ins-32B scores 84.9% avg (90.5 basic / 79.4 advanced); GTA1-7B at 83.4%.
  • ScreenSpot-Pro: UI-Ins-32B 57.0% avg (73.7 text / 30.0 icon); prior best SE-GUI-7B at 53.6%.
  • ScreenSpot-V2: UI-Ins-32B 94.9% avg; UI-Ins-7B 94.0%.
  • ShowDown: UI-Ins-32B 73.8%, UI-Ins-7B 73.1%.
  • AndroidWorld (online agent): UI-Ins-7B + GPT-5 planner achieves 74.1% success rate, surpassing Gemini 2.5 Computer Use (73.3%) and UI-TARS-2 (69.7%).
  • Cleaning data alone (no diversity augmentation) already yields consistent gains across all benchmarks (Fig. 2c).
  • Using only Qwen2.5-VL-7B zero-shot, diverse instruction perspectives give 76% relative gain over original instructions on ScreenSpot-Pro.

Limitations

  • AndroidWorld experiments require GPT-5 as the high-level planner; the grounding model itself is not end-to-end agentic.
  • Multi-perspective instruction augmentation via GPT-4.1 introduces pipeline cost and potential LLM hallucination despite verification.
  • Four predefined perspectives (appearance, functionality, location, intent) are heuristically chosen; coverage of edge cases is untested.
  • Evaluation on AndroidWorld uses a simple architecture without extra knowledge or domain-specific prompt engineering; headroom from richer scaffolding is unexplored.
  • Icon grounding remains substantially weaker than text grounding on ScreenSpot-Pro (30.0% vs. 73.7%), indicating visual element localization is still a hard sub-problem.

Relevance to Agentic AI / LLM Agents

GUI grounding is the low-level perception-action bottleneck for any GUI agent; UI-Ins directly addresses it by framing instruction understanding as structured reasoning, which generalizes to the chain-of-thought and tool-use patterns central to LLM agent design. The SFT+GRPO pipeline demonstrates how to use RL-from-environment-feedback (point-in-box binary reward) to teach agents to self-select analytical strategies — a transferable pattern for agents that must choose among multiple reasoning modalities. The 74.1% AndroidWorld success rate, achieved with a compact 7B grounding executor paired with a frontier planner, is relevant for modular agent architectures where specialized small models act as reliable action primitives. The finding that SFT on coordinate-only targets causes policy collapse in RL — and that diverse reasoning traces prevent it — is a practical insight for anyone fine-tuning action-prediction agents with GRPO or PPO.