LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension¶

🕒 Published (v1): 2026-07-02 12:32 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV egocentric video REC benchmark; tests long-form temporal grounding in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LongEgoRefer is a benchmark for Video Referring Expression Comprehension (Video REC) in long-form egocentric video, built from Ego4D footage averaging 45 minutes in duration with targets appearing in only 1.3% of frames. It exposes a severe performance gap in current VLMs and video grounding models when tasked with locating sparsely-occurring, interaction-centric objects in hour-scale first-person recordings. Even GPT-5 achieves only 37.3% tIoU@0.1 and 17.5% vIoU@0.1, underscoring the benchmark as largely unsolved.

Problem¶

Existing egocentric Video REC benchmarks (e.g., RefEgo at ~12 s clips, 75.5% object appearance rate; EgoMask at ~90 s, 60.3% appearance rate) use short, trimmed clips where target objects appear densely throughout the video. These settings fail to reflect real-world conditions: continuous hour-long recordings where objects appear transiently (sparse occurrences), natural language queries require disambiguation across visually similar temporal contexts, and interactions are dynamic and complex. No benchmark jointly demands long-form temporal search, interaction-centric linguistic grounding, and frame-level spatial localization.

Method¶

Benchmark construction: LongEgoRefer is built by mapping EgoTracks per-frame bounding-box annotations back to their parent full-length Ego4D videos (rather than the 6-minute clips EgoTracks uses). Filtering criteria require videos \(\geq\) 20 minutes, object tracks \(\geq\) 10 seconds where the wearer actively manipulates the object (is_active = True).

Referring expression generation: Gemini 2.5 Flash is prompted to produce 10 candidate captions per clip; these are aggregated into one cohesive expression. All expressions undergo a four-stage human curation pipeline—validation, correction, disambiguation (ensuring referential exclusivity across the full video), and quality assurance with a second annotator re-reviewing 52.2% of samples. In total 95.39% of Gemini-initialized descriptions were substantially revised; 25.1% of the 2,000 initial candidates were discarded, yielding 1,498 final samples.

Baselines: Two strategies are evaluated. (1) Sliding-window inference: existing short-clip Video REC models (VideoLISA, Sa2VA, Grounded SAM2, SAM3) applied to temporal windows; results are aggregated by highest confidence score. (2) Training-free two-stage pipeline: a VLM identifies the temporal interval ("when"), then Grounding DINO generates an initial bounding box on the middle frame, which SAM2 propagates into a full spatio-temporal track ("where").

Key Contributions¶

LongEgoRefer benchmark: 433 Ego4D videos (avg. 45 min), 1,498 referring expressions (avg. 63.5 words), 1.3% frame-level object appearance rate, with explicit human–object interaction (HOI) annotations.
Standardized baselines: sliding-window adaptation of existing Video REC models and a decoupled VLM + Grounded SAM2 two-stage pipeline as reference strategies for long-form evaluation.
Empirical analysis: identification of two fundamental failure modes—(1) sliding-window models lack global temporal awareness and accumulate false positives; (2) the two-stage pipeline suffers from error propagation, where temporal mislocalization strictly bounds spatial accuracy.

Results¶

All numbers from Table 2 on LongEgoRefer:

GPT-5 (best overall): tIoU@0.1 = 37.31, vIoU@0.1 = 17.48, mvIoU = 7.89
Gemini 2.5 Pro: tIoU@0.1 = 23.30, vIoU@0.1 = 12.01, mvIoU = 5.49
Gemini 2.5 Flash: tIoU@0.1 = 10.08, vIoU@0.1 = 4.49, mvIoU = 2.21
GPT-4o: tIoU@0.1 = 19.69, vIoU@0.1 = 10.68, mvIoU = 4.49
Best open-source (Qwen3-VL 32B): tIoU@0.1 = 4.87, mvIoU = 0.97
Best open-source 8B (Qwen3-VL 8B): tIoU@0.1 = 4.74, mvIoU = 0.83
Best Video REC model (Sa2VA): tIoU@0.1 = 0.60, mvIoU = 1.72
Under stricter metrics (mvIoU, mtIoU), all models score below 8%, confirming the task is largely unsolved even for frontier closed-source models.
Significant open-source vs. closed-source performance gap observed at all metric thresholds.

Limitations¶

Benchmark size: 1,498 samples from 433 videos; fine-tuning experiments or training splits are not provided.
Single continuous occurrence: the task is restricted to grounding one continuous appearance of a referred object; multiple or discontinuous occurrences are not evaluated.
Text truncation in paper: the Experiments section is cut off, so full ablation and qualitative analysis details are unavailable.
Language generation bias: although 95.39% of expressions were human-revised, the seed captions originate from Gemini 2.5 Flash, which may introduce systematic biases toward visual features that Gemini describes well.
No training split: LongEgoRefer is a test-only benchmark; there is no released training or fine-tuning set to facilitate model development directly on this distribution.
English only: linguistic diversity and cross-lingual grounding are not addressed.

Relevance to Vision-Language Models¶

LongEgoRefer directly stresses the temporal reasoning and long-context grounding capabilities of VLMs: the benchmark's primary baselines are large VLMs (Gemini family, GPT-4o, GPT-5) used for coarse temporal localization before spatial grounding, revealing that even frontier closed-source models fail badly at identifying rare events in hour-scale video. The stark open-source/closed-source performance gap (Qwen3-VL 32B mvIoU = 0.97 vs. GPT-5 mvIoU = 7.89) points to open questions about long-context video encoding, instruction following over extended sequences, and the VLM's ability to match fine-grained interaction descriptions to sparse visual evidence. For researchers tracking VLMs, this benchmark provides a concrete stress test distinguishing models on real-world temporal grounding, a capability underrepresented in current VLM evaluation suites.