Skip to content

See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm

🕒 Published (v1): 2025-12-09 14:14 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

See-Control introduces Embodied Smartphone Operation (ESO), a framework enabling a low-DoF robotic arm to physically interact with any smartphone using only screen captures and an MLLM — no ADB, no OS-level access. It delivers a 155-task benchmark, a GPT-4o-backed embodied agent, and a richly annotated dataset of operation episodes. The system treats the phone as a physical object, making it platform-agnostic and privacy-preserving by design.

Problem

All existing smartphone agents depend on Android Debug Bridge (ADB), which locks them to Android, requires developer/debug mode, routes sensitive data through software bridges, and is incompatible with home-robot deployment where a robot must physically pick up and operate any device. No benchmark or agent addressed the unique physical constraints of robotic single-contact-point smartphone interaction.

Method

The agent operates as a POMDP: at each step it receives a screenshot (via screen mirroring) plus task history, emits a chain-of-thought "thought action," then selects a discretized hardware action (Tap, Swipe, Text, Back, Exit) mapped to robotic arm coordinates. GUI navigation is split into two grounding paths: OCR for text elements (returning bounding boxes whose centers become tap targets) and Grounding DINO for icons (all candidate bboxes are indexed and overlaid as visual prompts; the MLLM selects the correct index, following Set-of-Mark). Back/Exit are handled by gesture-based swipes; text input uses a one-time anchor-point keyboard layout calibration to localize each key pixel-by-pixel. The GPT-4o backbone also performs self-reflection — when it detects an error state it backtracks and retries with modified parameters.

Key Contributions

  • First formal definition and benchmark for the Embodied Smartphone Operation (ESO) task: 155 tasks (37 standard, 103 challenging single-app, 15 cross-app) with difficulty tiers, golden-step annotations, and three metrics (SR, CR, SE).
  • See-Control agent: fully ADB-free, vision-only, platform-agnostic MLLM agent generating physical robotic arm control commands.
  • Annotated dataset of operation episodes with 10 fields per step (screenshot, instruction, observation, thought, abstract action, hardware action, GUI bboxes, bbox type, step correctness, final success) supporting VLA and PRM training.
  • User study (n=25) documenting user rejection of ADB-based agents on privacy/platform grounds and strong preference for embodied alternatives.

Results

  • Standard Task Set (single-app): Simple SR 11/13 (85%), Medium 5/13 (38%), Hard 1/11 (9%).
  • Challenging Task Set (single-app): Simple SR 23/32 (72%), Medium 18/35 (51%), Hard 3/36 (8%).
  • Cross-app tasks: SR 2/15 (13%), CR 0.333, SE 0.649.
  • Performance degrades stepwise with task complexity; CR and SE also drop as error accumulation compounds across longer sequences.
  • No ADB-based baseline is evaluated head-to-head; results establish a first-attempt baseline only.

Limitations

  • Single-contact-point arm constrains the action space to single-touch; multi-touch gestures (pinch, rotate, long-press, drag-and-drop) are unsupported.
  • MLLM inference latency causes execution errors on rapidly changing UIs (e.g., games).
  • Keyboard localization requires a one-time per-device calibration, reducing portability.
  • No comparison against ADB-based agents; results are a standalone baseline with no competitive numbers.
  • Small user study (n=25), no IRB or statistical analysis reported.

Relevance to Harnesses / Meta-Harnesses

See-Control is a concrete instantiation of a perception-to-action harness: it wraps an MLLM with a structured pipeline of specialized visual tools (OCR, Grounding DINO, SoM visual prompting), a discretized action executor, and a self-reflection loop — exactly the harness pattern where a meta-controller routes subtasks to specialist modules. The POMDP formulation with thought-action/env-action separation and history-conditioned self-correction mirrors the plan–act–verify loop common in meta-harness designs. The benchmark and dataset also serve as evaluation scaffolding for future harnesses targeting embodied GUI agents, and the stated roadmap toward end-to-end VLA integration anticipates replacing the modular harness with a tighter learned pipeline — a recurring architectural tension in this line of work.