Traj-VLN: Learning Pixel-Space Interaction via Autoregressive Trajectory Generation

🕒 Published (v1): 2026-07-12 12:54 UTC · Source: Arxiv · link

Why this paper was selected

Autoregressive pixel-space trajectory generation for VLN; novel grounding approach for navigation agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Traj-VLN fine-tunes VLMs to autoregressively generate sequences of 2D pixel coordinates that trace a navigation trajectory on overhead images, rather than predicting a single target pixel or discrete action chunks. This pixel-space trajectory supervision acts as a chain-of-thought (CoT) mechanism that improves pixel-goal prediction accuracy without requiring explicit 3D reasoning. Built on Qwen3-VL-8B, it achieves state-of-the-art success rates on VLN-CE benchmarks using less training data than competing methods.

Problem

VLMs used for Vision-and-Language Navigation in Continuous Environments (VLN-CE) must plan 3D spatial interactions, but are pretrained almost exclusively on 2D RGB images. Incorporating depth or geometric priors requires costly cross-modal alignment and still yields limited gains. Single pixel-goal supervision (the current best alternative) asks the model to infer a 3D-consistent target pixel without any intermediate spatial reasoning scaffold, limiting accuracy in cluttered environments.

Method

Traj-VLN is built on Qwen3-VL-8B with the vision encoder frozen; only the LLM and projection layer are fine-tuned. Navigation proceeds as a two-stage conversation per step:

1. Stage 1 (turn or stop): the VLM outputs a turning symbol (←, →) or STOP, or ↓ to proceed forward.
2. Stage 2 (trajectory generation): on receiving ↓, the agent tilts its camera 30° downward to capture an overhead view; the VLM then autoregressively generates a textual sequence of normalized pixel coordinates [u₁,v₁] [u₂,v₂] … [uₖ,vₖ] tracing a trajectory from the bottom-center of the overhead image outward, trained with standard cross-entropy loss.

Ground-truth trajectories are constructed by projecting future agent poses \(T_{i}\) relative to the current pose \(T_\text{cur}\) onto the overhead image plane via camera intrinsics \(K\) and extrinsics \(T_{c2b}\):

\[ (X_i,Y_i,Z_i,1)^T = d_i T_{c2b}^{-1} K^{-1}(u_i,v_i,1,\tfrac{1}{d_i})^T \]

Occluded and duplicate waypoints are filtered using the depth map. At inference, pixel coordinates are lifted back to 3D waypoints using the depth image for robot control. Training uses 1.75M samples from R2R-CE, RxR-CE, and a ScaleVLN subset, sampled at 4-step intervals along trajectories.

Key Contributions

• Proposes pixel-space trajectory supervision as an alternative to single pixel-goal or action-chunk supervision for VLM fine-tuning in VLN-CE.
• Demonstrates that autoregressive trajectory generation functions as a CoT mechanism: using the final trajectory point as the pixel-goal outperforms direct pixel-goal prediction under identical settings.
• Shows that trajectory supervision consistently outperforms pixel-goal supervision regardless of how many waypoints are executed per prediction step.
• Achieves SOTA success rate on both R2R-CE and RxR-CE val-unseen splits while using fewer training samples (1.75M) than prior leading methods (InternVLA-N1: 4.72M).

Results

Flagship model (Qwen3-VL-8B, frozen encoder, 1.75M samples) vs. SOTA on val-unseen: - R2R-CE: SR = 58.4, SPL = 51.7 — outperforms InternVLA-N1 (SR 55.4, SPL 52.1, 4.72M samples) and Goal2Pixel (SR 53.9, SPL 52.7) - RxR-CE: SR = 52.6, nDTW = 62.8 — outperforms InternVLA-N1 (SR 49.5, nDTW 62.6) - Surpasses JanusVLN by +5.6 SR (R2R-CE) and +1.2 SR (RxR-CE)

Controlled ablation (Qwen2.5-VL-7B, LoRA rank 32, identical data/config, 1.75M samples): - Pixel-goal supervision (InternVLA-N1-S2): R2R SR = 39.6, RxR SR = 36.09 - Traj-VLN final-pixel evaluation: R2R SR = 43.2 (+3.6), RxR SR = 41.54 (+5.45) - Traj-VLN trajectory evaluation: R2R SR = 43.6, RxR SR = 39.38 - Best configuration (1 waypoint executed per prediction): +4.0 SR on R2R-CE and +3.29 SR on RxR-CE over pixel-goal baseline

Limitations

• Requires a depth map at inference to lift pixel trajectories to 3D waypoints, limiting applicability to depth-sensor-equipped robots.
• Trajectory evaluation exhibits a trade-off: executing more waypoints per prediction lowers OSR, suggesting the full planned trajectory is not always safe to execute blindly.
• Trained for only one epoch; multi-epoch training or unfreezing the vision encoder may yield further gains but at substantially higher compute cost (unfreezing adds ~62.5% training overhead).
• Evaluation is simulation-only (R2R-CE/RxR-CE); real-world generalization is discussed qualitatively but not benchmarked quantitatively.
• Uses a single fixed robot height and camera angle setting for data collection, whereas InternVLA-N1 uses multiple configurations — partly explaining the sample-count difference.

Relevance to Vision-Language Models

This paper directly addresses the fundamental distributional mismatch between VLMs' 2D pretraining and the 3D spatial reasoning demands of embodied navigation — a core open problem for deploying general-purpose VLMs in robotics. The finding that autoregressive pixel-trajectory generation acts as a CoT mechanism is broadly relevant: it suggests that structuring fine-tuning supervision to match the sequential, compositional nature of VLM generation improves grounding even without architectural changes. For researchers tracking VLMs, the controlled ablation cleanly isolates supervision signal (trajectory vs. point) as the critical variable, providing actionable evidence for how to adapt VLMs to spatial tasks without 3D data. The result that freezing the vision encoder while fine-tuning only the LLM and projection layer is both computationally efficient and effective reinforces a widely-debated practical design choice in VLM adaptation.