Skip to content

GAIR: GUI Automation via Information-Joint Reasoning and Group Reflection

🕒 Published (v1): 2025-12-10 07:40 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GAIR is a multi-MLLM orchestration framework for GUI automation that routes GUI-specific models for perception and a general-purpose MLLM for reasoning/decision-making, augmented by a group reflection mechanism that re-tasks specialists when initial information is insufficient. Using only 7B-class models totaling ≤30B parameters, it surpasses 72B single-model baselines on two benchmarks.

Problem

No single MLLM covers the full breadth of GUI automation tasks (web, desktop, mobile, CAD, vertical applications): GUI-specialist models have narrow training distributions and weak general reasoning, while general-purpose MLLMs lack fine-grained GUI perception. Naively combining them without structured information integration leads to conflicting decisions and undetected errors.

Method

GAIR decomposes each GUI automation step into three stages:

  1. Observation: An ensemble of GUI-specific MLLMs (UI-TARS-7B-DPO, InfiGUI-R1-3B, UGround-V1-7B) independently process the GUI screenshot and task specification, each outputting structured textual observations.
  2. Reasoning & Decision: A general-purpose MLLM (Qwen2.5-VL-7B) ingests all specialist outputs via multi-turn dialogue, cross-validates them using world knowledge, and selects an operation.
  3. Group Reflection: If the general-purpose model judges its information insufficient for a confident decision, it enters a reflection state — generating model-specific instructions and hints tailored to each specialist's known strengths/weaknesses, triggering a second round of targeted information extraction before re-reasoning.

The whole pipeline is training-free at the framework level; all component models are used off-the-shelf.

Key Contributions

  • A heterogeneous multi-MLLM framework that separates GUI perception (specialists) from reasoning and decision (general-purpose model), connected via natural-language information passing.
  • A group reflection mechanism that enables the coordinator to adaptively re-task specialists with targeted instructions, functioning as an error-avoidance and correction loop.
  • Empirical evidence that ensemble orchestration of small (3–7B) models can outperform monolithic 72B models on GUI grounding benchmarks.
  • Ablation analysis quantifying each component's contribution (single model → multi-model → multi-model + reflection).

Results

  • UI-I2E-Bench: 78.4% overall success rate vs. UGround-V1-72B at 76.3% (prior SOTA); best on most platform/element subtypes; Input (+4.1%) and Toggle (+5.8%) over second-best.
  • ScreenSpot: 91.0% overall vs. UI-TARS-7B at 89.5% and UGround-V1-72B at 89.4%.
  • Ablation (UI-I2E-Bench): Single best model (UGround-V1-7B) = 70.3%; multi-model without reflection = 76.5%; full GAIR = 78.4%.
  • Information integration effectiveness: decision correct rate jumps from 56.8% (only 1 specialist correct) to 87.0% (2 specialists correct), well above random selection.

Limitations

  • Evaluated only on GUI grounding benchmarks (element location); end-to-end task completion on interactive environments (e.g., AndroidWorld) is not reported.
  • Vertical/domain-specific GUI automation (CAD, medical, etc.) untested and noted as future work; framework utility depends on availability of domain-specific specialist models.
  • No latency or computational cost analysis for the multi-model pipeline including reflection rounds.
  • Group reflection round count is not bounded or analyzed; potential for runaway inference cost unstated.
  • Training data distributions of specialists are not disclosed in full; coverage gaps are asserted qualitatively.

Relevance to Harnesses / Meta-Harnesses

GAIR is a textbook meta-harness pattern applied to GUI automation: a central coordinator (Qwen2.5-VL-7B) orchestrates specialist sub-agents, aggregates their outputs, and dynamically re-dispatches them with targeted sub-prompts when quality is insufficient — identical in structure to reflection-loop meta-harnesses used in code generation or research pipelines. The group reflection mechanism specifically instantiates the "completeness critic → re-task specialists" pattern common in multi-agent harnesses, demonstrating that meta-level control flow (rather than larger models) is a viable path to performance gains. For harness researchers, the key design choice here — natural-language as the inter-agent communication medium, coordinator-as-judge, per-specialist adaptive instructions — is a concrete implementation reference for heterogeneous-specialist orchestration.