Skip to content

FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Proactive and personalized mobile LLM agent benchmark; 20K realistic usage episodes

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FingerTip 20K is a benchmark of 21,437 real Android interaction episodes from 95 users, introducing two novel evaluation tracks: proactive task suggestion (inferring user intent without explicit instruction) and personalized task execution (adapting action sequences to user-specific preferences). The best models reach only 12.8% success on proactive suggestion vs. 30.3% for humans, revealing a large capability gap that existing mobile GUI benchmarks cannot measure.

Problem

Existing mobile GUI agent benchmarks (e.g., AitW, AndroidWorld) exclusively evaluate reactive execution of explicit instructions and provide no user-related contextual information (location, time, profile, history). This prevents measuring two important capabilities: (1) proactively predicting and suggesting tasks without an instruction, and (2) personalizing action trajectories to match individual user preferences rather than a generic "correct" solution.

Method

Data collection: 95 crowdsourced Android users ran a custom FingerTip APP on their personal phones for one month, recording real intents (one sentence), location category, and demonstrating the full action sequence (clicks, types, scrolls, etc.) via Android accessibility API. Each episode includes screenshots, accessibility tree XML, and structured action sequences. Up to 12 episodes/user/day were allowed, yielding 21,437 episodes across 506 apps with an average of 11.1 steps.

Proactive task suggestion (Track 1): Given user profile \(U\), timestamp \(T\), scenario \(S\), historical intents \(I_{\text{history}}\) (up to 20), and partial screenshots \(O\), the agent must output \(I = f(U, T, S, I_{\text{history}}, O)\) — a sentence predicting the user's current intent. Evaluation uses cosine embedding similarity \(S_1\) (paraphrase-multilingual-MiniLM-L12-v2) and Levenshtein similarity \(S_2\), averaged as \(\text{Sim}_1 = (S_1+S_2)/2\), plus binary success rate \(SR_1\) judged by DeepSeek-V3.

Personalized task execution (Track 2): Given \(U\), the true intent \(I_{\text{true}}\), historical action sequences \(A_{\text{history}}\) from a past similar task, and current state \((O_t, AT_t)\), the agent outputs \(A_{t+1}\) iteratively. Evaluation uses manual success rate \(SR_2\), a personalization score \(\text{Sim}_2 = S_I / S_{II}\) (ratio of Levenshtein similarity to same-user golden actions vs. different-type-user golden actions), and step ratio. Physical Android phones were used via ADB to avoid emulator restrictions on Chinese apps.

Key Contributions

  • FingerTip 20K benchmark: 21,437 episodes, 506 apps, 95 real users, with user profile, timestamp, location, and long-term history — the first mobile GUI benchmark with all three contextual dimensions.
  • Two new evaluation tracks (proactive task suggestion, personalized task execution) with corresponding metrics (\(SR_1\), \(\text{Sim}_1\), \(SR_2\), \(\text{Sim}_2\), step ratio).
  • Empirical validation that user action-sequence preferences are measurable: Levenshtein similarity is significantly higher within same user/type than across types.
  • Comprehensive evaluation of 11+ models (GPT-4.1, Qwen-QVQ-Max, UI-TARS-1.5-7B, AppAgent, etc.) plus a human study on 400 episodes.
  • Fine-tuned Qwen-2.5-VL-7B via LoRA on collected data, demonstrating gains from user-oriented training data.

Results

Proactive task suggestion (Track 1): - Best model: Qwen-QVQ-Max, \(SR_1 = 12.8\%\), \(\text{Sim}_1 = 0.39\) - GPT-4.1: \(SR_1 = 7.2\%\), \(\text{Sim}_1 = 0.35\) - Human annotators (20 people, 400 episodes): \(SR_1 = 30.3\%\), \(\text{Sim}_1 = 0.57\) - Human–machine gap: \(>2\times\) on success rate

Personalized task execution (Track 2): - Best overall (GUI-specific): UI-TARS-1.5-7B, \(SR_2 = 38.5\%\), step ratio 1.22 - Best among prompt-engineering agents: AppAgent (GPT-4.1), \(SR_2 = 11.0\%\), best \(\text{Sim}_2 = 1.12\), step ratio 1.13 - Generalist models (GPT-4.1, Qwen-VL-Max): \(SR_2 \leq 5.5\%\), hampered by poor GUI grounding - \(\text{Sim}_2 \approx 1\) for all models, indicating no model successfully personalizes action trajectories to individual users

Fine-tuning (Track 1, Qwen-2.5-VL-7B, LoRA): - Full training set fine-tune with rank 64 yielded the largest gains (exact numbers truncated in provided text) - More screenshots as input \(O\) consistently raise \(SR_1\) across all models

Limitations

  • Human study uses annotators distinct from the original users, who lack the personal context of those users, so the 30.3% human ceiling may underestimate true human performance.
  • Data collection was limited to 95 users in China over one month, raising questions about cultural and linguistic generalizability and long-term coverage.
  • Physical phone evaluation via ADB introduces setup complexity and potential hardware variability; scaling to larger test sets is expensive.
  • \(\text{Sim}_2 \approx 1\) across all models indicates the benchmark's personalization track is currently unsolvable by existing approaches; no model demonstrates genuine preference learning.
  • Fine-tuning results are partially truncated in the provided text; full ablation numbers are not available for analysis.
  • The \(SR_1\) binary judgment relies on DeepSeek-V3, which may not perfectly capture semantic equivalence of user intents.

Relevance to Agentic AI / LLM Agents

This work directly advances the frontier of proactive and personalized LLM agents — two properties increasingly viewed as essential for autonomous agents embedded in personal workflows. The proactive suggestion track formalizes the problem of intent inference from context without explicit instruction, which is a prerequisite for truly autonomous agents that reduce user cognitive load. The personalization track challenges agents to go beyond task success and match individual behavioral patterns, a requirement that current GUI-control architectures (even fine-tuned ones) systematically fail. For researchers building long-horizon agentic systems, FingerTip 20K provides the first large-scale, real-world evaluation surface that includes temporal user history and contextual signals — bridging the gap between single-session benchmarks and the continuous, user-specific nature of real deployment.