Skip to content

MobileIPL: Enhancing Mobile Agents Thinking Process via Iterative Preference Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Iterative preference learning improves mobile VLM agent chain-of-action reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MobileIPL trains VLM-based mobile GUI agents by constructing a Chain of Action-Planning Thoughts (CoaT) tree via MCTS-style iterative sampling, then using rule-based rewards back-propagated through the tree to form Thinking-level DPO (T-DPO) preference pairs—without a learned process reward model. A three-stage instruction evolution via GPT-4o mitigates SFT overfitting and improves reasoning diversity. The resulting 7B agent achieves state-of-the-art step accuracy on AITZ, AMEX, and AndroidControl, outperforming continual-pretraining models OS-Atlas and UI-TARS.

Problem

VLM-based mobile GUI agents benefit from intermediate reasoning (CoaT), but: (1) SFT on fixed CoaT trajectories causes overfitting to static reasoning patterns; (2) self-training methods that reward only final answers ignore intermediate step quality, leading to reward hacking; (3) process reward models (PRMs) that assess individual steps require expensive manual step-level annotations, which is especially burdensome in GUI environments requiring real devices or simulators.

Method

Warm-up SFT with instruction evolution. Three levels of Q&A are synthesized from real mobile UI screenshots using GPT-4o: (L1) basic grounding/description tasks, (L2) widget caption and relational structure, (L3) advanced GUI FAQ. These are mixed with CoaT trajectories and used for warm-up SFT to expand the diversity of the seed model's output distribution.

CoaT-tree construction. Each action \(\hat{a}_i\) is decomposed into a four-step multi-turn dialogue \([s_1, s_2, s_3, s_4]\) = (description, thought, action-decision, grounding). At each step \(t\), \(K\) continuations are sampled to form a tree. Leaf-node rewards are computed rule-based:

\[v(s_t) = v_{\text{type}} + \text{score}_{\text{match}}\]

where \(\text{score}_{\text{match}}\) uses normalized coordinate distance for CLICK actions and F1 for INPUT actions. Intermediate node values are back-propagated as discounted averages of child values:

\[v(s_{t-1}) = c \cdot \frac{1}{K}\sum_{k=1}^{K} v(s_t^{(k)})\]

T-DPO training. Trees are classified into \(\alpha\) (all correct), \(\beta\) (mixed), \(\gamma\) (all incorrect). From \(\beta\), contrastive pairs \((s_t^{(k)\uparrow}, s_t^{(k')\downarrow})\) are extracted when their value difference exceeds \(1/K\); from \(\gamma\), the ground-truth action is paired against sampled failures. DPO is applied at the thinking-step level rather than the full trajectory:

\[\mathcal{L}_{\text{T-DPO}} = -\mathbb{E}\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(s_t^+|s_{1:t-1})}{\pi_{\text{ref}}(s_t^+|s_{1:t-1})} - \beta\log\frac{\pi_\theta(s_t^-|s_{1:t-1})}{\pi_{\text{ref}}(s_t^-|s_{1:t-1})}\right)\right]\]

The updated model is re-used as the sampling policy for further iterations.

Key Contributions

  • MCTS-inspired CoaT-tree construction with rule-based leaf rewards and backward credit assignment, replacing learned PRMs.
  • Thinking-level DPO (T-DPO) that optimizes intermediate reasoning steps rather than full trajectories.
  • Three-stage instruction evolution using GPT-4o to increase CoaT diversity and prevent SFT overfitting.
  • Demonstrated SOTA on AITZ, AMEX, and AndroidControl with a 7B model, surpassing continual-pretraining models trained on orders-of-magnitude more GUI data.

Results

  • AITZ (total match): MobileIPL 69.15% vs. seed model 55.40% (+13.75%), vs. OS-Atlas-7B 65.11% (+4.04%), vs. UI-TARS-7B 65.61% (+3.54%), vs. Falcon-UI-7B 69.10% (+0.05% despite Falcon-UI pre-trained on 3M GUIs).
  • AMEX (overall): MobileIPL 74.29% vs. SphAgent-7B 70.71% (+3.58%), vs. OS-Atlas-7B 70.33% (+3.96%), vs. UI-TARS-7B 70.33% (+3.96%).
  • AndroidControl Step.Acc: MobileIPL 72.7% vs. UI-TARS 72.5% (+0.2%), vs. OS-Atlas 71.2% (+1.5%), vs. Qwen2-VL(SFT) 69.1% (+3.6%).
  • OOD (AndroidControl app-unseen/task-unseen): MobileIPL 70.0%/72.2% vs. OS-Atlas 60.7%/66.2%—smaller OOD performance drop than pre-trained baselines.
  • Ablation (AITZ R1): Removing IPL: −5.0%; removing instruction evolution: −2.5%; removing negative samples: −4.0%; replacing T-DPO with naive DPO on full trajectory: −5.1%.
  • Low-resource: 1/2 training data (R1) loses only 0.6%; 1/5 data (R2) loses 4.8% vs. full-data R1.

Limitations

  • The CoaT paradigm decomposes actions into four fixed turns; the multi-turn overhead increases inference latency and token cost relative to single-step agents.
  • PRESS action accuracy is notably lower than other action types (23.5% in R1), indicating the method does not yet generalize uniformly across all action categories.
  • Instruction evolution data requires GPT-4o inference plus human filtering, introducing external closed-model dependency and curation cost.
  • Evaluation is restricted to mobile Android GUI environments; generalization to web or desktop GUI domains is not assessed.
  • Data efficiency comparison is against SFT and naive DPO, not against online RL methods at matched compute budgets.

Relevance to Vision-Language Models

MobileIPL is a concrete demonstration of applying preference optimization to VLM-based embodied agents where environment feedback is expensive to obtain, directly relevant to the broader challenge of aligning VLMs for interactive decision-making. The key insight—that MCTS-style tree search with rule-based leaf rewards can replace learned PRMs for step-level credit assignment—extends naturally to other GUI or agentic VLM settings. The instruction evolution technique addresses a recurring pain point in fine-tuning VLMs on narrow domain data: distribution collapse and reasoning rigidity. This work contributes to the growing line of GUI-agent research (OS-Atlas, UI-TARS, AriaUI) that treats the VLM not merely as a perception backbone but as a reasoning engine operating over structured visual contexts.