Skip to content

Lightweight Neural App Control

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LiMAC (Lightweight Multi-modal App Control) is a gated mobile app control architecture that routes most actions through a compact 520M-parameter Action Transformer (AcT) and invokes a fine-tuned VLM only for text-generation actions (<15% of steps). On two Android benchmarks, LiMAC achieves up to 19% higher accuracy than standalone fine-tuned VLMs and up to 42% higher than GPT-4o prompt-engineering baselines, while running ~30× faster (≈3 s/task vs. ~60–120 s for GPT-4o agents).

Problem

Large VLMs and GPT-4o-based app agents are computationally prohibitive on mobile hardware: a GPT-4o agent costs ~$1/task and takes 1–2 minutes per task. Existing small fine-tuned VLMs improve speed but still underperform because they must handle all action types (clicks, scrolls, text entry) with a single auto-regressive model, which is inefficient for the majority of actions that do not require natural language generation.

Method

LiMAC decomposes app control into two branches:

  1. Action Transformer (AcT) — a 520M-parameter transformer that receives as input a sequence of episode embeddings: a BERT-encoded goal, per-UI-element embeddings (concatenation of fine-tuned CLIP image embedding, BERT text embedding, and attribute embedding with positional encoding), and previous action embeddings. AcT predicts the action type via cross-entropy classification. For click actions, it predicts the target UI element using a contrastive InfoNCE objective: a learned projection aligns the transformer's hidden state with the correct UI element embedding via cosine similarity over all UI elements in the episode as negatives.

  2. Fine-tuned VLM branch — activated only when AcT predicts input-text or open-app. The predicted action type is used to constrain the VLM's decoding prefix (e.g., {"action-type":"input-text","text":), and the VLM completes the specification. Two VLMs are evaluated: Florence2 (820M, full fine-tune) and Qwen2-VL (2B, LoRA).

CLIP is fine-tuned on app-control screenshots using InfoNCE to align image patches with their UI-tree text descriptions before being frozen inside AcT.

Key Contributions

  • LiMAC architecture: gated hybrid of AcT + small fine-tuned VLM that separates action-type/click prediction from text-generation.
  • AcT module: efficient sequence transformer for action-type classification and click-target prediction without auto-regressive decoding.
  • Contrastive click targeting: InfoNCE-based contrastive objective over UI element embeddings within an episode, avoiding fixed-class classification.
  • Fine-tuned small VLMs (Florence2 and Qwen2-VL at ≤2B parameters) that match or exceed GPT-4o on text-generation actions.
  • Demonstrated robustness to missing or imprecise UI trees (text embeddings removal degrades overall accuracy only marginally; visual features are primary).

Results

  • Overall accuracy vs. fine-tuned VLMs: LiMAC with Florence2 achieves 72.2% / 63.1% (AitW / AndroidControl) vs. Florence2 alone at 70.8% / 57.0%; LiMAC with Qwen2-VL achieves 70.9% / 62.5% vs. Qwen2-VL alone at 51.0% / 52.2%.
  • Overall accuracy vs. GPT-4o baselines: M3A (best GPT-4o baseline) scores 35.6% / 57.5%; LiMAC exceeds it by up to ~35 pp on AitW.
  • Inference time: LiMAC with Florence2 averages 0.34 s/step; LiMAC with Qwen2-VL 0.63 s/step; best GPT-4o baseline (M3A) averages 10.64 s/step.
  • Action-type accuracy: AcT achieves 86.9% on AitW and 82.3% on AndroidControl, outperforming all baselines.
  • Click-target accuracy (AitW): AcT reaches 77.4%, compared to 48.3% for M3A and 76.2% for Florence2 alone.
  • Ablation: removing image embeddings drops overall accuracy from 63.1% to 56.0% on AndroidControl; removing CLIP fine-tuning drops it to 60.0%; removing text embeddings is nearly lossless (63.0%).

Limitations

  • Trained on small datasets (13K episodes for AndroidControl, 18K for AitW) with no pretraining of AcT, limiting generalization to complex or unseen tasks.
  • Higher memory footprint than single-model approaches because both AcT and a VLM must be loaded simultaneously.
  • No online interaction or error recovery; evaluated only on offline benchmark datasets with no success-rate metric for full task completion.
  • Safety and privacy concerns (credit card data, personal identifiers) in mobile settings are unaddressed.
  • VLM branch is called for <15% of actions but dominates latency when triggered; complex tasks with frequent text-entry will erode the speed advantage.

Relevance to Vision-Language Models

LiMAC demonstrates a practical decomposition strategy where a small VLM is used selectively—only for language-intensive sub-tasks—rather than as a universal action generator, which is a key insight for deploying VLMs in resource-constrained agentic settings. The paper shows that fine-tuned sub-2B VLMs (Florence2, Qwen2-VL) can match or exceed GPT-4o on constrained text-generation sub-tasks when the action-type routing is handled by a non-auto-regressive module. The contrastive UI-element grounding objective offers a reusable technique for grounding VLM-adjacent models to structured visual UIs without bounding-box regression. This work extends the broader trend of modular VLM agent architectures where vision encoders (CLIP) are domain-adapted via contrastive fine-tuning to serve as perception backbones for downstream decision-making.