Xiaomi-GUI-0 Technical Report

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Industry-scale GUI agent from Xiaomi; directly applicable to agent building

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Xiaomi-GUI-0 is a native end-to-end multimodal GUI agent for real mobile environments, trained and evaluated within a closed loop on physical devices rather than emulators or static benchmarks. It combines a real-device-dominant hybrid infrastructure, an error-driven data flywheel, and a three-stage training pipeline (SFT → Step RL → Agentic RL) to achieve 72.0% task success on the new RealMobile benchmark and 78.9% on AndroidWorld.

Problem

Existing mobile GUI agents are trained on clean offline trajectories from emulators or simulated environments and evaluated on static benchmarks, creating a persistent distribution gap with real-world deployment. Real applications expose dynamic states—captchas, payment authentication, anti-emulator detection, permission dialogs, and risk-control mechanisms—that simulated setups cannot faithfully reproduce, causing high benchmark scores to poorly predict actual usability.

Method

The system rests on three interconnected components:

Real-device-dominant hybrid infrastructure: Hundreds of physical phones and tablets (covering ~100 commercial apps) form the primary execution substrate; sandbox instances handle apps with low risk-control requirements. A Device-Pull scheduling scheme lets idle devices claim tasks only when their readiness profile matches task requirements, avoiding wasted execution on devices in degraded states.

Multi-source training data with an error-driven flywheel: Three data tiers address different coverage needs—high-frequency task data (annotated head-function trajectories with 14 abnormal-state types, ~5,000 abnormal samples); high-generalization data (five-level function trees, behavior-bucket query synthesis, rollout-based trajectory collection with trajectory- and step-level cleaning); and agent-capability enhancement data (chain-of-thought synthesis into a five-field schema: observation, reflection, planning, decision, memory). The flywheel converts failure trajectories from real rollouts into corrective training signal: human annotators replay failures and label the critical erroneous step with the correct action and error rationale, while a teacher model scoring/takeover mechanism generates recovery demonstrations when the student model repeatedly scores low.

Three-stage progressive training: 1. SFT: Establishes baseline UI operation and canonical task paths. 2. Step RL: Applies group-relative reinforcement learning with step-level reward signals to optimize per-step action selection, state assessment, and local correction. 3. Agentic RL: Runs in real/near-real mobile environments; optimizes long-horizon planning, state memory, reflective error correction, and recovery-oriented execution.

Intermediate-page data augmentation covers three scenarios: continuation mid-path, re-navigation from an off-path page, and cross-application page confusion resolution. Per-step foreground application identity (via ADB package-name mapping) is annotated to disambiguate visually similar pages across apps.

Key Contributions

• Real-device-dominant hybrid infrastructure spanning smartphones, tablets, and in-vehicle cockpits, with Device-Pull scheduling for physical device state management.
• Error-driven data flywheel that targets the model's own failure distribution via interactive annotation of erroneous steps and teacher-model recovery demonstrations, producing reflection, correction, and recovery training data.
• RealMobile benchmark: constructed from real user traffic, executed on physical devices against live applications, with fine-grained sub-goal scoring and cross-application scenarios.
• Three-stage training pipeline (SFT → Step RL → Agentic RL) that incrementally develops basic execution, per-step optimization, and long-horizon error recovery.

Results

• RealMobile (real-device, live apps): 72.0% end-to-end task success rate (Xiaomi-GUI-0).
• AndroidWorld (simulated benchmark): 78.9% success rate.
• Competitive UI element grounding on ScreenSpot-V2 grounding benchmark.
• Substantially improved execution stability and abnormal-state recognition compared to prior baselines (paper reports comparative improvement but specific baseline numbers are in Tables not fully included in the provided text).

Limitations

• The physical device pool and its maintenance (account warm-up, login recovery, risk-control cooldowns) impose operational overhead that is not easily replicated outside a large commercial lab setting.
• Coverage is bounded by enrolled devices and applications; the 100-app phone coverage and 20-app tablet/cockpit coverage may not generalize to the full long tail of commercial apps.
• Anti-emulator detection forces reliance on physical hardware, making large-scale ablation studies costly; sandboxes are restricted to apps with mild risk-control.
• RealMobile is an in-house benchmark tied to Xiaomi's device ecosystem, limiting reproducibility for external researchers.
• The error-driven flywheel requires human annotators to replay failures and label erroneous steps, introducing annotation cost and potential labeler inconsistency.
• The base VLM architecture and parameter count are not specified in the provided text, making direct capability comparisons opaque.

Relevance to Vision-Language Models

Xiaomi-GUI-0 directly operationalizes VLMs as sequential decision-making agents in dynamic real-world GUI environments, extending VLM research from static visual grounding toward closed-loop agentic control with error recovery. The three-stage training pipeline—particularly Step RL with group-relative rewards and Agentic RL in live environments—represents a concrete recipe for post-training VLMs for long-horizon action generation beyond single-turn visual question answering. The error-driven flywheel's use of teacher-model takeover and reflective chain-of-thought supervision is directly applicable to broader VLM fine-tuning strategies targeting robustness and self-correction. For researchers tracking VLMs, the RealMobile benchmark and the distribution-gap analysis between emulator-based and real-device evaluation constitute an important methodological contribution challenging how GUI-agent VLMs are assessed.