Skip to content

ReferTrack: Referring Then Tracking for Embodied Visual Tracking

🕒 Published (v1): 2026-07-22 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReferTrack reformulates embodied visual tracking (EVT) as a two-step "referring then tracking" pipeline: a VLA model first selects the target pedestrian by emitting a single index token over image-space bounding boxes, then predicts waypoints conditioned on that grounded decision. A sliding-window queue of selected bounding boxes injects target motion geometry into visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. On EVT-Bench, a 4B-parameter SFT-only model achieves state-of-the-art single-view performance, matching or exceeding multi-camera baselines on identification-heavy tasks.

Problem

Prior VLA-based EVT methods (e.g., TrackVLA++) perform chain-of-thought reasoning in abstract spatial latent spaces (polar-coordinate tokens) that are hard to supervise and weakly aligned with image-space visual grounding. In crowded or ambiguous scenes, this abstraction gap causes identification failures that neither scaling model size nor adding cameras easily compensates for.

Method

ReferTrack is a dual-branch VLA built on Qwen3-4B. At each timestep \(T\):

  1. Candidate catalog construction: YOLO11+ByteTrack detects pedestrians; sorted by bbox area into \(C_T = \{\langle\text{ped}_1\rangle, \ldots, \langle\text{ped}_K\rangle, \langle\text{NO\_EXIST}\rangle\}\). Each entry is encoded by a shared two-layer MLP \(P_\text{bbox}\).

  2. Refer-CoT token: A first LLM forward pass over language tokens \(E_L\), catalog \(C_T\), and structured visual history \(E^V_{1:T}\) emits a single token \(E^{\text{refer}}_T \in \{\langle\text{ped}_k\rangle\} \cup \{\langle\text{NO\_EXIST}\rangle\}\) — a constrained classification over the indexed catalog.

  3. TVBI encoding: Historical frame tokens are interleaved with temporal-viewpoint-bbox indicator (TVBI) tokens: \(E_\text{TVBI}(t) = E_\text{TVI}(t) + P_\text{bbox}(b_t)\), where \(b_t \in [0,1]^4\) is the previously selected target bbox (or \([0,0,0,0]\) sentinel for absence). The current frame deliberately receives only TVI (no bbox injection), forcing the model to ground the target from history alone.

  4. Trajectory prediction: A second LLM pass conditioned on \(E^{\text{refer}}_T\) produces an action token \(E^A_T\), decoded by an MLP head into \(M\) waypoints \(W_T = \{(x_i, y_i, \theta_i)\}\).

Training objective: \(\mathcal{L} = \alpha\mathcal{L}_\text{traj} + \mathcal{L}_\text{refer} + \mathcal{L}_\text{text}\), with \(\alpha=10\). \(\mathcal{L}_\text{traj}\) is MSE over waypoints; \(\mathcal{L}_\text{refer}\) is cross-entropy over the Refer-CoT token; \(\mathcal{L}_\text{text}\) is cross-entropy over Refer-QA text outputs. Co-training uses 1.3M navigation trajectories (Habitat 3.0 / EVT-Bench) and 1.3M Refer-QA samples synthesized from SYNTH-PEDES at a 1:1 ratio, with two-stage SFT (vision projector alignment, then full fine-tuning).

Key Contributions

  • Referring-then-tracking paradigm: reformulates EVT target identification as a constrained single-token index selection over image-space bboxes, enabling direct supervision and VLM-native grounding.
  • TVBI tokens: extend TVI (temporal-viewpoint indicator) from NavFoM with target bbox geometry, creating a target-conditioned spatiotemporal memory for trajectory planning.
  • Refer-QA co-training: a custom dataset derived from SYNTH-PEDES that transfers indexed-bbox selection capability from static images to dynamic online tracking without additional closed-loop data.
  • State-of-the-art single-view EVT: 4B SFT-only model that matches or exceeds multi-camera baselines and RL-refined methods on EVT-Bench identification-heavy splits.
  • Sim-to-real transfer: validated on Unitree Go2 (quadruped) and Unitree G1 (humanoid) at 10.6 Hz inference with 12 ms detection latency.

Results

All numbers from EVT-Bench (SR / TR / CR); single forward-view setting unless noted.

  • ReferTrack (Ours, 4B, SFT): STT 89.4 / 92.5 / 1.6; DT 73.3 / 81.8 / 7.6; AT 74.1 / 85.7 / 7.7
  • TrackVLA++ (7B, no RL) — strongest prior single-view baseline: STT 86.0 / 81.0 / 2.10; DT 66.5 / 68.8 / 4.71; AT 51.2 / 63.4 / 15.9
  • ReferTrack gains: +6.8 SR / +13.0 TR on DT; +22.9 SR / +22.3 TR on AT
  • VLingNav (7B, RL-refined): STT 88.4 / 81.2 / 2.1; DT – ; AT –; ReferTrack surpasses on DT SR without RL
  • Multi-camera references (not ranked against): TrackVLA++ (7B, multi-cam) DT 55.9/63.8/15.1, AT 15.7/35.7/53.3 — ReferTrack single-camera surpasses both
  • Ablation (DT split):
  • Oracle TVBI (GT bboxes, no Refer-CoT): 81.5 SR / 84.7 TR — upper bound showing identification is the primary bottleneck
  • w/o Refer-CoT & TVBI: 55.7 SR / 71.4 TR (−17.6 / −10.4 vs full model)
  • w/o TVBI only: 70.4 SR / 80.8 TR (−2.9 / −1.0)

Limitations

  • Target identification remains the primary bottleneck in distracted scenarios (oracle TVBI yields +8.2 SR over the full model), indicating room for improvement in the Refer-CoT step.
  • Catalog size is capped at \(K\) candidates; beyond \(K\) pedestrians, lower-priority (smaller bbox area) detections are discarded — potentially losing the true target in dense crowds.
  • Relies on an off-the-shelf detector (YOLO11+ByteTrack); detector failures propagate directly into the catalog, with no fallback.
  • Real-world evaluation is qualitative only; no quantitative metrics for sim-to-real gap are reported.
  • Current-frame fine tokens receive no bbox injection by design — if the historical queue is corrupted (e.g., sustained occlusion), recovery depends entirely on raw visual features.

Relevance to Vision-Language Models

ReferTrack demonstrates a principled way to inject structured spatial grounding into VLM-based action policies: rather than letting the LLM hallucinate or reason in latent spatial codes, it converts target identification into a constrained vocabulary classification over image-space region proposals — a design directly aligned with how VLMs naturally process bbox-centric representations. The Refer-QA co-training strategy shows that large-scale static referring/grounding data (which the web provides abundantly) can be leveraged to bootstrap embodied tracking without costly closed-loop rollouts, a broadly applicable recipe for data-efficient VLA training. For researchers tracking VLMs, this work is a concrete instance of grounding CoT reasoning in verifiable image-space evidence — a theme increasingly central to multimodal reasoning, referring expression comprehension, and spatial VLM capabilities.