Skip to content

OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OWMM-Agent is a VLM-based agent for Open-World Mobile Manipulation (OWMM) that maintains multi-view scene frames and robot state memory for long-horizon decision-making, issuing high-level actions via function calls to classical planners. To overcome domain shift from pre-trained VLMs, the authors build an agentic data synthesis pipeline that generates 572K CoT-annotated simulation episodes and fine-tunes InternVL-2.5 into OWMM-VLM. The resulting 38B model achieves 21.90% full-task success in simulation versus 0.33% for GPT-4o+PIVOT, and 90% single-step success on a real Fetch robot.

Problem

Pre-trained VLMs suffer severe domain shift when applied directly to OWMM: (1) they fail at non-object-centric affordance grounding (e.g., detecting navigable areas), (2) they cannot track robot state across long horizons, and (3) they lack embodiment priors (e.g., kinematic reach constraints). Prior 3D semantic map approaches (SLAM + CLIP embeddings) are limited by embedding model capacity and computationally expensive for dynamic, open-ended environments.

Method

Architecture (OWMM-Agent): A unified VLM (InternVL-2.5 fine-tuned) takes as input a language instruction, a pre-mapped pose graph with associated RGB images, the current egocentric RGB+depth frame, and a textual robot history summary. It produces high-level actions in JSON—image retrieval ID, navigate-to-point coordinate (bounding box center), pick bounding box, or place bounding box—via chain-of-thought reasoning. A path planner (coverage planning) and motion planner (sampling-based) convert these to low-level joint/base commands; 2D-to-3D back-projection maps predicted bounding boxes to 3D targets using depth.

CoT design: Each inference step outputs: (1) task instruction reasoning, (2) perceptual grounding of ego and scene images, (3) action decision with affordance coordinates, (4) step summarization fed as history to the next step. This prevents dead loops and supports temporal state tracking.

Data synthesis pipeline (4 stages): 1. PDDL-based task planning in Habitat simulator (143 HSSD scenes, 157 objects, 1,471 receptacles, 400 episodes/scene). 2. Trajectory execution with keyframe candidate marking (no dense RGB collection). 3. Strategic keyframe selection: visibility + reachability filters for pick/place; interval sampling for navigation. 4. Template-based CoT annotation with GPT-4o mini paraphrasing for linguistic diversity.

Final dataset: 21,046 valid episodes, ~572K annotations (Pick: 64.7K, Place: 68.9K, Nav-to-point: 59.6K, Search scene frame: 378.8K).

Key Contributions

  • OWMM-Agent: unified multi-modal agent with long-term image memory (pose graph) + transient text state memory, driving a Fetch robot via function-called classical planners.
  • Agentic data synthesis pipeline producing large-scale, CoT-annotated OWMM episodes from simulation with minimal human annotation.
  • OWMM-VLM: first dedicated foundation model for mobile manipulation supporting multi-image reasoning, robot state tracking, and executable multi-modal action generation in one model.
  • Structured CoT framework that eliminates dead loops and substantially reduces hallucination compared to base VLMs.

Results

Single-step evaluation (simulation): - Egocentric decision-making: OWMM-VLM-38B 97.85% vs. GPT-4o 48.53%, InternVL2.5-8B 17.52% - Image retrieval: 87.54% vs. GPT-4o 46.46%, InternVL2.5-8B 1.27% - Affordance grounding (object): 0.97 ± 0.14 vs. GPT-4o 0.56, GPT-4o+PIVOT 0.67 - Affordance grounding (receptacle): 0.94 ± 0.19 vs. GPT-4o 0.35, GPT-4o+PIVOT 0.45 - Affordance grounding (navigation): 0.88 ± 0.17 vs. GPT-4o 0.07, GPT-4o+PIVOT 0.05

Episodic evaluation (simulation, strict 0.85m threshold): - Full task success: OWMM-VLM-38B 21.90% vs. GPT-4o+PIVOT 0.33%, GPT-4o+Robopoint 0.33% - Dead loops: OWMM-VLM-38B 0/308 vs. GPT-4o+PIVOT 195/308

Real-world (single-step, Fetch robot): - OWMM-VLM-38B: total accuracy 90% vs. GPT-4o+PIVOT/Robopoint 46.67% - Zero-shot sim-to-real: model trained only on synthetic data

Limitations

  • Requires a pre-mapping phase (SLAM + pose graph construction) before task execution, limiting fully autonomous deployment in novel environments.
  • Episodic full-task success remains low (21.90%) in simulation, indicating failure accumulation over long horizons.
  • Real-world evaluation uses human confirmation before each action execution for safety—no fully autonomous real-robot episodic evaluation reported.
  • Training data is entirely simulation-derived (Habitat/HSSD); real-world generalization may degrade in more diverse or cluttered scenes.
  • Cross-embodiment transferability is unaddressed; the system is tailored to the Fetch robot's kinematic priors.

Relevance to Vision-Language Models

This paper demonstrates a concrete path to adapting general-purpose VLMs (InternVL-2.5) into task-specific embodied agents through targeted supervised fine-tuning on synthetically generated, CoT-annotated data—directly relevant to VLM domain adaptation methodology. The structured CoT design for hallucination suppression in multi-image, long-horizon settings addresses a known VLM failure mode (multi-image hallucination, dead loops) with practical engineering rather than architectural changes. For VLM researchers, the single-step benchmarks (image retrieval, affordance grounding) expose specific capability gaps in frontier models like GPT-4o when confronted with egocentric spatial reasoning and non-object-centric grounding. The agentic data synthesis approach generalizes as a blueprint for generating instruction-tuning datasets in domains where human annotation is impractical.