Exploratory, Communicative, and Deployable: Vision-Driven Embodied Agents for Open-World Mobile Manipulation¶

🕒 Published (v1): 2026-07-15 09:55 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

REAL is a sim-to-real-consistent agentic framework for open-world mobile manipulation that removes oracle perceptual APIs and adds a simulated user for human-in-the-loop intent disambiguation. It fine-tunes Qwen3-VL-8B via SFT followed by GSPO-based online RL within a privilege-free visual environment. The resulting agent achieves 56.9% success on interactive simulation tasks and 78.3% end-to-end success across 60 real-world dual-arm robot episodes with zero-shot transfer.

Problem¶

Existing embodied agent benchmarks either (1) grant agents oracle perceptual APIs (global object lists, teleport-to-target tools, privileged 3D poses) that cannot be reproduced on real hardware, causing sim-to-real failures; or (2) assume unambiguous, complete user instructions and open-loop execution, ignoring the common real-world need for active intent disambiguation through multi-turn dialogue. Neither challenge—oracle-free visual exploration nor communicative interaction—has been jointly addressed at the training level for VLM-based mobile manipulation.

Method¶

Environment (REAL). Built on GRUtopia, the environment exposes only a receptacle-level structural prior (IDs and semantic categories) obtainable from a robot's mapping stack. The agent explores via a four-tool deployable chain: nav_to (inter-receptacle navigation), walk_around (open-vocabulary 2D scanning around a receptacle), show_object_by_category (ego-centric category isolation), and gaze_at (approach and center target). Detected objects are back-projected from 2D segmentation masks into 3D and highlighted with visual IDs in RGB/SoM observations, enabling manipulation without privileged coordinates. A Gemini-3-flash-powered simulated user issues deliberately ambiguous instructions, accesses the agent's action-observation history, and answers ask queries in context.

Training pipeline. 1. SFT data synthesis: A rule-based planner generates expert trajectories; a Social Annotation phase uses LLMs to vague-ify instructions and inject ask/response pairs. Historical states \(h_t = (\Sigma_t, \text{task}_t)\) are annotated by Gemini-3-pro, forming dataset \(\mathcal{D} = \{(T^{(i)}, \tau^{(i)})\}\). 2. SFT (behavior cloning): Qwen3-VL-8B-Instruct is fine-tuned on \(\mathcal{D}\) with frozen vision encoder by minimizing \(\mathcal{L}_\text{SFT} = -\sum_{(T,\tau)\in\mathcal{D}}\sum_{t=1}^L \log \pi_\theta(a_t, h_t \mid T, o_t)\). 3. Online RL with GSPO: Group Sequence Policy Optimization replaces GRPO's step-wise importance ratios with a length-normalized sequence ratio, compressing variance by \(1/L_i^2\) to stabilize long-horizon rollouts. Rewards are World-Graph-differential (\(\delta_t = \text{WG}_t - \text{WG}_{t-1}\) matched against \(\Delta_\text{task}\)), with per-step timing penalty \(-\lambda_\text{time}\), severity-scaled failure penalties, and a difficulty-conditioned query budget \(B_\text{ask}\) rewarding appropriate ask usage.

Key Contributions¶

REAL framework: Closed-loop visual environment with zero oracle perceptual APIs, physically obtainable receptacle priors, and a Gemini-powered simulated user for dynamic multi-turn intent alignment.
Training pipeline: Automated task-generation and social-annotation pipeline; dual-stage SFT + GSPO online RL on Qwen3-VL-8B enabling tool-use alignment and adaptive replanning.
REAL-Bench: 241-task benchmark spanning active exploration, visual distraction, articulated manipulation, and interactive disambiguation.
Real-world deployment: Zero-shot transfer to a physical LIFT2 dual-arm mobile robot, achieving 78.3% end-to-end success over 60 episodes.

Results¶

REAL agent achieves 56.9% success rate on interactive disambiguation tasks in simulation, outperforming leading commercial closed-source VLMs (baselines unnamed in the excerpt but described as "leading commercial closed-source VLMs").
78.3% end-to-end success rate over 60 real-world episodes on a dual-arm mobile robot, with zero-shot transfer to unseen household scenarios.
REAL-Bench covers 241 tasks across 4 families; baselines include PARTNR, EMMOE, ERA, OWMM-Agent, and commercial VLM APIs (exact per-baseline numbers not in the provided text excerpt).

Limitations¶

The physical toolchain relies on a pre-built 3D occupancy map and receptacle semantic segmentation (from prior scans); environments where such maps are unavailable are not addressed.
The simulated user is powered by a closed-source model (Gemini-3-flash), introducing a dependency that complicates fully open or offline evaluation.
Task scope is restricted to cross-receptacle object rearrangement; generalization to tasks requiring different manipulation primitives or longer semantic chains is not demonstrated.
Only 60 real-world episodes are reported; statistical significance and coverage of diverse failure modes remain limited.
The RL reward budget \(B_\text{ask}\) requires manual difficulty-conditioned tuning, which may not transfer automatically to new task distributions.

Relevance to Vision-Language Models¶

REAL directly benchmarks and fine-tunes VLMs (Qwen3-VL-8B) in embodied settings that demand visual grounding, long-horizon reasoning, and interactive disambiguation—capabilities central to the VLM research agenda. The paper demonstrates that SFT + online RL can specialize a general-purpose VLM into a deployable robotic planner while preserving open-vocabulary reasoning, which informs work on VLM adaptation and tool-use alignment. The GSPO training objective and POMDP formulation with self-summarized history offer transferable techniques for any VLM application requiring multi-step decision-making under partial observability. REAL-Bench also provides a new eval surface for VLMs that goes beyond passive image-question-answering toward active, interactive, visually grounded task completion.