Skip to content

Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation

๐Ÿ•’ Published (v1): 2025-01-01 ยท Source: NeurIPS ยท Venue: NeurIPS 2025 ยท link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

Aux-Think addresses Test-time Reasoning Collapse (TRC) in Vision-Language Navigation (VLN), where explicit Chain-of-Thought (CoT) reasoning at test time degrades navigation performance compared to direct action prediction. The framework decouples CoT into auxiliary training tasks while keeping test-time inference free of explicit reasoning, achieving state-of-the-art success rate with significantly less training data. A 320K CoT-annotated VLN dataset (R2R-CoT-320k) is released to support this paradigm.

Problem

CoT reasoning strategies (Pre-Think, Post-Think) hurt VLN performance rather than help it โ€” a phenomenon the authors term Test-time Reasoning Collapse. This occurs because models trained only on oracle trajectories encounter out-of-distribution states at test time, causing CoT hallucinations that cascade into navigation failures across long-horizon multi-step trajectories. Prior VLN work using large pretrained models either ignores reasoning entirely or requires massive training datasets (2โ€“5M+ samples) for competitive performance.

Method

Aux-Think co-trains three tasks on the same VLM backbone (NVILA-lite 8B: SigLIP encoder + Qwen2 LLM), switching between them via prompt conditioning:

  1. CoT-based Reasoning (auxiliary): Given current + historical observations and instruction, predict step-wise reasoning traces c*_t. Loss: L_CoT.
  2. Instruction-based Reasoning (auxiliary): Given the full observation sequence, reconstruct the navigation instruction I_ฯ„ (reverse grounding). Loss: L_Ins.
  3. Receding-Horizon Action Planning (primary): Predict the next n actions (optimally n=3) given current context. At test time, only the first predicted action is executed. Loss: L_Act.

Total loss: L = ฮฃ_ฯ„ [L_Act_ฯ„ + L_CoT_ฯ„ + L_Ins_ฯ„].

At test time, CoT generation is suppressed entirely; the model acts via the internalized reasoning patterns without explicit chain generation, matching No-Think inference speed (1.25s/step vs. ~30s for Pre/Post-Think).

The R2R-CoT-320k dataset is constructed by prompting Qwen-2.5-VL-72B to generate step-wise reasoning traces for R2R-CE trajectories in Habitat simulator, producing 320K diverse CoT-annotated navigation samples.

Key Contributions

  • Test-time Reasoning Collapse finding: First systematic comparison of No-Think, Pre-Think, Post-Think for VLN-CE, revealing that both explicit CoT strategies consistently underperform direct action prediction.
  • Aux-Think framework: CoT as auxiliary training signal with No-Think testing; decouples reasoning internalization from execution, eliminating hallucination-driven action errors.
  • Receding-Horizon Action Planning: Multi-step action prediction (n=3 optimal) as the primary task, improving long-horizon stability.
  • R2R-CoT-320k dataset: First CoT-annotated dataset for VLN, covering 320K step-level reasoning traces grounded in R2R-CE trajectories.

Results

  • R2R-CE Val-Unseen (with extra data, 1.6M total): SR=54.8%, SPL=46.9% โ€” matches NaVILA (SR=54.0%, SPL=49.0%, 2.77M data) and Uni-NaVid (SR=47.0%, 5.57M data).
  • R2R-CE Val-Unseen (R2R-CoT-320k only, no extra data): Aux-Think SR=46.0%, SPL=40.5% vs. NaVid (SR=24.7%, SPL=23.6%) โ€” nearly 2ร— SR improvement at equal data.
  • RxR-CE Val-Unseen (1.92M data): SR=52.2% vs. NaVILA 49.3% (3.1M) and Uni-NaVid 48.7% (5.9M).
  • Reasoning strategy ablation (R2R-CoT-320k only): Aux-Think SR=41.3% vs. No-Think 35.1%, Pre-Think 11.4%, Post-Think 29.0%; inference time 1.25s vs. ~30s for explicit CoT.
  • Ablation (components): Full A+B+C (SR=46.0%) > A+B (44.2%) > A+C (38.9%) > A only (41.3%) > baseline No-Think (35.1%).
  • Aux-Think maintains SR on trajectories >70 steps where Pre/Post-Think collapse to near zero.

Limitations

  • Evaluated only under SFT on monocular RGB R2R-CE; no depth, panoramic, or localization signals tested.
  • Instruction-based reasoning (reverse task) and CoT generation require curated step-level annotations, which the authors generate via a large proprietary VLM (Qwen-2.5-VL-72B) โ€” annotation cost not discussed.
  • NE and SPL on RxR-CE are modest relative to SR gains, suggesting internalized reasoning biases the agent toward broader exploration and longer paths.
  • Receding-horizon planning degrades beyond n=3 steps due to limited perceptual field of monocular input; does not generalize well to longer planning horizons without richer sensing.
  • DAgger coverage of off-distribution states remains limited even for baseline comparisons.

Relevance to Vision-Language Models

Aux-Think directly addresses how to integrate CoT reasoning into VLMs deployed in dynamic, embodied settings โ€” a non-trivial transfer from static VQA/captioning contexts where Pre-Think excels. The Test-time Reasoning Collapse finding is an important negative result for the VLM community: explicit intermediate reasoning can harm sequential decision tasks, suggesting that reasoning must be internalized rather than emitted. The auxiliary supervision paradigm (CoT as training signal, suppressed at inference) is a practically reusable pattern for other action-centric VLM applications such as robotic manipulation, embodied QA, and GUI agents. R2R-CoT-320k also serves as a resource for studying reasoning grounding in visually-rich navigation environments.