Skip to content

OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OWMM-Agent is a VLM-based agent architecture for open-world mobile manipulation that maintains multi-view scene memory and robot state, generates chain-of-thought reasoning, and calls coordinate-based planners via function calling. The authors also introduce a simulation-based agentic data synthesis pipeline to adapt the VLM to the embodied domain. The fine-tuned OWMM-VLM-38B achieves SOTA across all single-step and episodic metrics, outperforming GPT-4o-based pipelines by large margins.

Problem

Existing OWMM approaches either rely on 3D semantic maps (limited by embedding model capacity, slow dense reconstruction) or directly apply general-purpose VLMs that suffer severe domain shift: low affordance grounding accuracy (0.05–0.18), near-zero multi-image retrieval accuracy (1.27% for InternVL2.5-8B), and dead loops from inability to track robot state across long horizons.

Method

The agent formulates OWMM as a multi-turn, multi-image, multi-modal reasoning problem over: (1) a pre-mapped camera pose graph with associated RGB images, (2) a live egocentric RGBD frame, and (3) a textual robot history maintained by the VLM itself. The VLM (fine-tuned InternVL-2.5) generates chain-of-thought reasoning then outputs one of four high-level action types—posed-image retrieval, navigate-to-point, pick, or place—with bounding box coordinates. A classical path planner and motion planner convert these to low-level joint/base commands via 2D-to-3D back-projection.

For domain adaptation, a four-stage agentic data synthesis pipeline is used: (1) PDDL task planning in Habitat simulation, (2) trajectory execution with keyframe candidate marking, (3) keyframe filtering by visibility/reachability criteria, (4) CoT annotation generation via templates with GPT-4o-mini paraphrasing for linguistic diversity. This yields ~572K total annotations across 21,046 episodes from 143 HSSD scenes.

Key Contributions

  • OWMM-Agent: a unified VLM agent framework with multi-view long-term memory, transient robot state tracking, and function-calling-based control—no separate perception modules at inference time.
  • OWMM-VLM: the first dedicated foundation model for mobile manipulation, fine-tuned on InternVL-2.5 (8B and 38B variants), supporting multi-image reasoning and multi-modal (bounding-box grounded) action generation.
  • Simulation-based agentic data synthesis pipeline producing 572K instruction-following episodes with structured CoT annotations, minimizing human annotation effort.
  • Structured CoT reasoning design targeting three failure modes: erroneous grounding, multi-image hallucination, and dead loops from insufficient state tracking.

Results

Single-step (simulation): - OWMM-VLM-38B: ego-centric decision-making 97.85% vs. GPT-4o 48.53%; image retrieval 87.54% vs. GPT-4o 46.46% - Affordance grounding (object): 0.97 vs. GPT-4o 0.56; (receptacle): 0.94 vs. GPT-4o 0.35; (navigation): 0.88 vs. GPT-4o 0.07 - Inference time 36.58s (38B) vs. GPT-4o 160.74s

Episodic (simulation, strict 0.85m threshold): - Full task success: OWMM-VLM-38B 21.90%, GPT-4o+PIVOT 0.33%, GPT-4o+Robopoint 0.33% - Dead loops: 0/308 for OWMM-VLM vs. 195/308 (GPT-4o+PIVOT) and 184/308 (GPT-4o+Robopoint)

Real-world (Fetch robot, human-verified single-step): - OWMM-VLM-38B total accuracy 90% (27/30 actions), GPT-4o+PIVOT/Robopoint 46.67% - Navigation affordance grounding: 10/10 (38B) vs. 0/10 for both GPT-4o pipelines

Limitations

  • Requires a pre-mapping phase (SLAM + pose graph construction) before task execution; not suitable for fully unknown environments without prior mapping.
  • Real-world episodic evaluation required human confirmation of each action for safety; no fully autonomous real-world end-to-end episodes reported.
  • 38B model has 36.58s per-step inference latency, limiting real-time reactivity.
  • Sim-to-real transfer validated only in a single lab environment; broader generalization across diverse real-world settings is untested.
  • Dataset diversity is limited to 143 HSSD scenes; ablations suggest scene/object diversity matters less than data volume for egocentric spatial intelligence, but this may not hold at deployment scale.

Relevance to Agentic AI / LLM Agents

This paper is a concrete instantiation of the VLM-as-agent paradigm in a physically grounded, long-horizon setting—where the VLM must maintain its own state memory, select tools (planners/controllers) via function calling, and generate structured multi-modal outputs rather than text. The agentic data synthesis pipeline—using PDDL planning + simulation to auto-generate CoT-annotated trajectories—is a scalable recipe for bootstrapping domain-adapted agents without human labeling, directly relevant to agent training methodology. The structured CoT design (perception → decision → action → summarize-for-next-step) provides a working template for state-tracking in tool-using agents. The dead-loop elimination results quantitatively demonstrate that explicit state summarization in the context window is critical for long-horizon agentic reliability.