Skip to content

NavBench: Probing Multimodal Large Language Models for Embodied Navigation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NavBench is a zero-shot benchmark for evaluating MLLMs on embodied navigation, decomposed into (1) navigation comprehension across three cognitively grounded QA tasks and (2) step-by-step execution across 432 episodes stratified by spatial, cognitive, and execution complexity. Evaluation of both proprietary and open-source models reveals that temporal reasoning (progress estimation) is a persistent bottleneck, comprehension and execution performance are tightly coupled, and lightweight open-source models can approach proprietary performance on easy tasks.

Problem

Existing embodied navigation benchmarks (R2R, ObjectNav) were designed for fully supervised agents and collapse evaluation to final success rate, providing no insight into whether a model genuinely understands navigation behavior. Static MLLM benchmarks do not test sequential decision-making. No prior benchmark jointly covers instruction comprehension, step-by-step execution, fine-grained difficulty stratification, and real-world deployability for zero-shot MLLMs.

Method

NavBench comprises two evaluation tracks built on Matterport3D simulator data augmented with panoramic and single-viewpoint RGB images:

Navigation Comprehension — 3,200 multiple-choice QA pairs across three tasks: - Global Instruction Alignment (1,200 items): match a panoramic trajectory to one of five candidate instructions; distractors use four perturbation strategies (random, directional replacement, object replacement, sub-instruction shuffle). - Temporal Progress Estimation (1,000 items): given a truncated trajectory and the full sub-instruction list, identify the last completed sub-instruction. - Local Observation-Action Reasoning (1,000 items): Future-Observation Prediction (current view + action → next view) and Future-Action Prediction (two consecutive views → action).

Navigation Execution — 432 episodes across 72 scenes; at each step the MLLM selects from candidate navigable viewpoints given the instruction and panoramic observation. Episodes are stratified into easy/medium/hard via composite complexity scores over three orthogonal dimensions: - Spatial complexity: log path length, turn-angle std, elevation change, 2D area (Eq. 2). - Cognitive complexity: instruction length, verb/spatial-term/landmark/clause counts (Eq. 3). - Execution complexity: step count, turns, floor changes, decision points (Eq. 4). Scores are log-normalized to [1,9] and validated by human raters.

A modular real-world pipeline maps MLLM viewpoint selections to robot motion via a ResNet-50 waypoint predictor, MLLM decision module, and low-level controller deployed on a dual-arm mobile robot with Intel RealSense D435.

Key Contributions

  • NavBench benchmark: zero-shot MLLM evaluation decomposed into comprehension (3 tasks, 3,200 QA pairs) and execution (432 episodes, 72 scenes).
  • Three-axis difficulty taxonomy (spatial, cognitive, execution) with automatic scoring + human validation.
  • Real-world robot deployment pipeline connecting MLLM outputs to low-level motion commands.
  • Systematic evaluation of 10 MLLMs (3 closed, 7 open-source) with ablations on map context, CoT prompting, and distractor type.

Results

  • GPT-4o achieves best execution average (SR 81.59% easy / 41.67% medium / 27.78% hard; exec avg 41.33%) among closed models; o4-mini leads comprehension avg (59.66%).
  • Qwen2.5-VL-7B is the strongest open-source model (comprehension avg 45.26%, exec avg 21.77%), approaching GPT-4o-mini (46.42%, 27.99%).
  • Progress Estimation is the weakest subtask across all models; only GPT-4o exceeds 40% (42.90%); most others remain near or below random.
  • InternVL2.5-2B achieves 67.25% on Global Instruction Alignment, surpassing GPT-4o (51.33%), but collapses on harder subtasks.
  • All models fail under sub-instruction shuffle, confirming temporal ordering as a critical weakness.
  • Map context (topological text prompts) improves GPT-4o SR by +4.51 pp at medium difficulty, less at easy/hard.
  • CoT ("Let's think step by step") improves GPT-4o Global Alignment by +9.09% but provides marginal or negative gains elsewhere.
  • GPT-4o error rate on Progress Estimation rises from 35.3% (short paths) to 76.1% (long paths).
  • Real-world pilot: GPT-4o 60% SR, Qwen2.5-VL-7B 40% SR on 10 indoor test cases each.

Limitations

  • Execution evaluation uses viewpoint selection (abstracted discrete actions) rather than continuous low-level control, which may not fully reflect real deployment difficulty.
  • Real-world validation is limited to a 10-case pilot in a controlled indoor lab; generalization to diverse or unstructured environments is untested.
  • Benchmark is built exclusively on Matterport3D indoor scenes; outdoor or other environment types are not covered.
  • Difficulty weights (α, β, γ) are set empirically without principled optimization.
  • CoT ablations use only the generic "Let's think step by step" prompt; more structured reasoning prompts are not explored.

Relevance to Vision-Language Models

NavBench directly probes whether MLLM visual-linguistic reasoning transfers to embodied sequential decision-making, a key open question as VLMs are increasingly deployed in robotics. The comprehension–execution correlation finding suggests that VLM reasoning quality predicts downstream navigation policy quality, motivating improvement of VLM temporal and spatial understanding as a proxy for embodied capability. The benchmark's diagnostic decomposition (global/progress/local) provides a finer-grained signal than end-to-end success rates, revealing temporal reasoning as the dominant bottleneck not visible in aggregate metrics. The zero-shot evaluation protocol and real-world pipeline make NavBench directly relevant for researchers evaluating VLMs as general-purpose robot controllers without task-specific fine-tuning.