Xiaomi-GUI-0 Technical Report

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Xiaomi-GUI-0 is a native end-to-end multimodal GUI agent for real Android mobile environments, trained and evaluated in a real-device closed loop rather than emulators or static benchmarks. It introduces a real-device-dominant hybrid infrastructure, an error-driven data flywheel that converts failure trajectories into correction supervision, and a three-stage training pipeline (SFT → Step RL → Agentic RL). It achieves 72.0% on the authors' real-device benchmark RealMobile and 78.9% on AndroidWorld.

Problem

Existing mobile GUI agents train and evaluate primarily on offline success trajectories, emulated environments, and static benchmarks. These settings fail to capture: (1) anti-emulator detection that blocks production apps in virtual environments; (2) real abnormal states (captchas, payment auth, risk-control dialogs, expired sessions); (3) dynamic account states, asynchronous page loads, and continuously evolving app versions; (4) long-tail user intents and tasks initiated from arbitrary intermediate pages. The result is a persistent gap between high benchmark scores and real-world execution stability.

Method

Real-device-dominant hybrid infrastructure. Hundreds of physical phones and tablets (covering ~100 commercial apps) serve as the primary execution substrate; sandbox instances handle apps where emulation is stable. A Device-Pull scheduling scheme lets idle devices claim tasks from a queue based on current readiness (ADB connectivity, account state, risk-control level), avoiding assignment to devices that go offline mid-task.

Multi-source training data in three tiers: - High-frequency task data: expert-annotated trajectories for head functions across real devices, augmented for intermediate-page starts (mid-path, off-path, cross-app) and ~5,000 abnormal-state samples across 14 anomaly types. - High-generalization data: a five-level function tree built by hierarchical app exploration → behavior-bucket query synthesis → rollout on hybrid infra → trajectory- and step-level cleaning, extending coverage to long-tail intents. - Agent-capability enhancement data: structured chain-of-thought synthesis normalizing reasoning into a five-field schema (observation, reflection, planning, decision, memory).

Error-driven data flywheel. Failed rollout trajectories are processed two ways: (i) interactive annotation identifies the critical erroneous step and provides the correct action, error category, and correction rationale; (ii) a teacher model scores student rollouts and takes over when quality is consistently low (e.g., repetitive actions, path deviation), generating recovery demonstrations from erroneous states back to correct paths. This produces error, reflection, and recovery data aligned to the model's own failure distribution.

Three-stage training pipeline: 1. SFT: establishes stable foundation for common app functions and canonical task paths. 2. Step RL: group-relative policy optimization with step-level reward signals targeting action selection, state assessment, and local error correction. 3. Agentic RL: conducted in real/near-real environments to improve long-horizon planning, state memory, reflective error correction, and recovery-oriented execution.

RealMobile benchmark. Constructed from real user traffic, executed on physical devices against live apps, scored via fine-grained sub-goal decomposition with executable verification rules and an auto-eval pipeline; covers cross-application scenarios and abnormal-state handling.

Key Contributions

• Real-device-dominant hybrid infrastructure spanning smartphones, tablets, and in-vehicle cockpits, enabling trajectory collection under real accounts, networks, and app risk-control mechanisms.
• Error-driven data flywheel that targets the agent's own failure distribution through interactive annotation (error localization + rationale) and teacher-model takeover (recovery demonstrations), producing reflection and correction supervision beyond mere volume scaling.
• Three-stage progressive training pipeline (SFT → Step RL → Agentic RL) that incrementally builds basic UI operation, fine-grained step-level decision quality, and long-horizon agentic execution.
• RealMobile: a real-device benchmark evaluated on live apps with fine-grained sub-goal scoring, executable verification rules, and cross-application task scenarios.

Results

• RealMobile (in-house real-device benchmark): 72.0% end-to-end task success rate.
• AndroidWorld (simulated task-execution benchmark): 78.9% task success rate.
• Competitive performance on static grounding benchmarks (ScreenSpot-V2) for UI element localization.
• Improved execution stability and abnormal-state recognition relative to prior approaches on real-device tasks (specific comparative numbers for baselines on RealMobile are not detailed in the provided text excerpt).

Limitations

• The hybrid infrastructure requires maintaining a large pool of physical devices with real accounts, making the setup expensive and difficult to replicate outside a device manufacturer's ecosystem (Xiaomi-specific hardware and accounts).
• Abnormal states requiring human participation (e.g., captchas) are handled by stopping execution and returning control to the user, rather than autonomous resolution—limiting full end-to-end automation.
• Anti-emulator detection means critical app behaviors remain difficult to reproduce at scale outside physical devices, constraining community reproducibility.
• Benchmark (RealMobile) and training data are constructed from Xiaomi's own app ecosystem and user traffic, which may limit generalizability to other device ecosystems or regional app distributions.
• The provided text does not detail comparisons against specific baseline models on RealMobile, making it hard to assess the individual contribution of each pipeline component from this excerpt.

Relevance to Agentic AI / LLM Agents

Xiaomi-GUI-0 directly advances real-world deployability of VLM-based agents—a core challenge in agentic AI—by demonstrating that train/eval distribution alignment with deployment conditions (real devices, real failure states) is as important as model architecture. The error-driven flywheel, which converts the agent's own rollout failures into targeted correction supervision, is a concrete instantiation of the self-improvement loop that agentic AI research increasingly targets. The three-stage RL pipeline (Step RL for local decision quality, Agentic RL for long-horizon recovery) illustrates a principled curriculum for grounding LLM agents in interactive environments. This work connects to the broader line of research on error recovery, reflective agents, and closing the sim-to-real gap for autonomous agents operating in unconstrained environments.