Skip to content

Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current egocentric video-language models (EgoVLMs) fail at a simple test: selecting the correct caption from candidates differing by a single verb or noun substitution. This paper introduces EgoHOIBench to quantify this failure and proposes EgoNCE++, an asymmetric contrastive objective that generates LLM/vocabulary-based hard negatives for video-to-text learning while preserving object-centric clustering via text-to-video positive sampling. EgoNCE++ yields consistent improvements across three EgoVLMs and seven downstream benchmarks.

Problem

Existing EgoVLMs pretrained with InfoNCE and EgoNCE lack fine-grained negative supervision: easy negatives (e.g., "cut grass" vs. "pick apple") train models to distinguish simultaneous verb-noun changes but not single-word variations. Additionally, both objectives implicitly create an object-centric feature space (noun clusters tighter than verb clusters), causing HOI-verb recognition to lag HOI-noun recognition. No prior benchmark isolated these failure modes.

Method

EgoHOIBench: A 29K-trial multiple-choice benchmark built from Ego4D where each trial requires selecting the correct caption from N LLM-generated distractors that differ only in the HOI verb (HOI-verb task) or HOI noun (HOI-noun task).

EgoNCE++ (asymmetric contrastive objective): - Video-to-text (V2T): Appends hard negative captions to the denominator of the contrastive loss. Negatives are generated either via LLM (LLaMA3-8B prompted to substitute HOI verbs/nouns with semantically different words) or via vocab-based substitution (random replacement from spaCy-extracted Ego4D vocabulary). Loss: L_v2t = log[exp(v·t) / (Σ_easy exp(v·t_j) + Σ_hard exp(v·t_k))]. - Text-to-video (T2V): Replaces EgoNCE's mixed verb/noun positive set with noun-only positives P_noun(v_i), explicitly reinforcing object-centric clustering without relying on additional visual data. - Training: Text encoder frozen (except word embeddings); visual encoder fine-tuned with LoRA (rank=alpha=16) for 10 epochs on EgoClip-3.8M (2.5M entries) using 8× A800 GPUs.

Key Contributions

  • EgoHOIBench: First benchmark specifically probing EgoVLMs' sensitivity to single-word HOI substitutions (verb vs. noun), revealing a large performance gap in all tested models.
  • EgoNCE++: Asymmetric contrastive loss that independently targets verb-level discrimination (V2T hard negatives) and noun-level clustering (T2V object-centric positives).
  • Diagnostic analysis: t-SNE visualization demonstrating that EgoVLMs inherently develop object-centric feature spaces, explaining the verb/noun performance asymmetry.
  • Demonstrated generalization across EgoVLP, EgoVLPv2, and LaViLa with zero-shot improvements on 7 benchmarks.

Results

  • EgoHOIBench (LaViLa→LaViLa++): +26.32% overall HOI-action accuracy; verb accuracy +34.02%.
  • ActionBench (temporal understanding, zero-shot): LaViLa 79.89% → LaViLa++ 91.18%, surpassing prior SoTA (InternVideo* 90.1%) and approaching human level (92.0%).
  • EK100-MIR (multi-instance retrieval): LaViLa++ avg. mAP 32.0% vs. LaViLa 30.8%; avg. nDCG 34.8% vs. 33.7%; surpasses HelpingHands (+37M params) and HENASY (+112M params) with no extra parameters.
  • EGTEA (action recognition): LaViLa++ mean-acc 34.0% vs. LaViLa 30.9% (+3.1%), best among compared methods.
  • EK100-CLS: +4.53% improvement over LaViLa baseline.
  • Ablation: LLM-based negatives outperform vocab-based, which in turn outperforms rule-based (BLEU-selected) by ≥+7.44% HOI-action; combining V2T + T2V yields +1.32% additional HOI-action gain over V2T alone.

Limitations

  • Pretraining is continued from existing EgoVLM checkpoints rather than from scratch, limiting assessment of EgoNCE++ as a standalone objective.
  • Hard negative generation with LLaMA3-8B adds inference overhead; the vocab-based alternative occasionally produces semantically meaningless combinations.
  • EgoHOIBench distractors are LLM-generated and thus may not fully capture the hardest real-world confounders.
  • Evaluation confined to Ego4D-derived and kitchen-domain datasets; generalization to diverse egocentric scenarios (medical, industrial) is untested.
  • The T2V noun-clustering loss may still underserve verb discrimination for highly ambiguous HOI verbs not well-covered by the pretraining vocabulary.

Relevance to Vision-Language Models

This paper directly probes a systematic blind spot in video-language contrastive pretraining—compositional sensitivity to fine-grained word substitutions—which mirrors analogous findings in image VLMs (e.g., CLIP's noun bias). The asymmetric loss design (separate objectives for hard negative discrimination vs. feature space geometry) offers a transferable blueprint for any VLM domain where noun and verb representations have different inductive biases. For researchers tracking VLMs, EgoHOIBench provides a concrete, reproducible diagnostic, and EgoNCE++ demonstrates that targeted contrastive objective engineering—without architectural changes—can close substantial performance gaps across multiple downstream tasks.