Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Gaze-VLM introduces a gaze-regularized attention mechanism that aligns VLM attention maps with human eye gaze distributions during training only, improving egocentric activity understanding and future event prediction without requiring gaze data at inference. The method adds a KL-divergence loss between transformer attention weights and temporally aggregated, occlusion-filtered gaze heatmaps. Across five VLM architectures and two tasks, it consistently yields 5–11% semantic score improvements.

Problem

VLMs applied to egocentric video (activity understanding, future event prediction) lack grounding in human visual attention; their attention mechanisms attend to task-irrelevant regions, degrading fine-grained prediction and producing hallucinations. Prior gaze-integration methods either use gaze as an inference-time input (limiting deployability) or are restricted to single architectures and non-generative tasks.

Method

A modular gaze-regularized attention block is inserted between the visual encoder and language decoder of any transformer-based VLM. During training: 1. Gaze heatmap construction: raw point gaze coordinates are Gaussian-smoothed into spatial heatmaps per frame. 2. Temporal aggregation with occlusion filtering: heatmaps from a δ=200ms window are warped to the current frame via optical flow and weighted by a bidirectional optical flow occlusion mask (oτ), producing a robust aggregated heatmap Hₜ. 3. Patch-wise alignment: Hₜ is pooled into patch-level scores H̃ₜ matching the ViT patch grid. 4. KL-divergence regularization: the training loss is L_total = L_CE + λ · Σₜ DKL(Aₜ ‖ H̃ₜ), where Aₜ are softmax attention weights from the model. A global query (aggregated across the input sequence rather than frame-specific) attends over spatial patches, and its output is compared against H̃ₜ.

At inference, no gaze or optical flow is required; only RGB frames are consumed.

Key Contributions

• Modular gaze-regularized attention loss (KL divergence) compatible with any attention-based VLM architecture.
• Occlusion-aware temporal gaze aggregation using bidirectional optical flow consistency to filter unreliable gaze points.
• Training-only gaze supervision paradigm: no gaze dependency at inference time.
• Comprehensive evaluation on Ego4D (in-distribution) and EGTEA+ Gaze (out-of-distribution) across five architectures (OpenFlamingo, Modified OpenFlamingo, LaViLa Narrator, InternVL-2.5-1B, OpenLLaVA-7B).
• Human evaluation demonstrating reduced visual hallucination rate (CI score 0.205 → 0.140).

Results

• Future event prediction (Ego4D, τₐ=2s): absolute gains of +8.9% to +10.5% Sentence-BERT semantic similarity across all five architectures over RGB-only baselines.
• Activity understanding (Ego4D): gains of +4.9% to +6.9%.
• Out-of-distribution (EGTEA+ Gaze, zero fine-tuning): +4.9% to +8.8% future prediction; +2.8% to +6.1% activity understanding.
• Longer horizon (τₐ=5s): gains maintained (e.g., OpenFlamingo 0.6297 → 0.7315) despite lower absolute scores.
• Attention-gaze alignment: top-10 patch overlap improves from 42% (base) to 68% (gaze-regularized).
• Runtime: inference time increases marginally (1.7s → 2.3s on NVIDIA A800); training occlusion filtering adds overhead but is absent at inference.
• Optimal regularization: λ=100 best across tasks; λ=1000 slightly underperforms; λ=0 consistently worst.

Limitations

• Gaze annotations are required during training; large-scale, high-quality calibrated gaze datasets are scarce, limiting scale.
• Evaluation metric (Sentence-BERT semantic similarity) may not fully capture factual accuracy or spatial precision of predictions.
• Hallucination evaluation is based on a small human study (200 samples, single dataset) rather than established benchmarks, since standard hallucination benchmarks (e.g., HallusionBench) lack gaze annotations.
• Optimal λ may need per-architecture and per-task tuning; the search range explored is coarse.
• Tasks and data are restricted to egocentric video; generalization to third-person or static-image VLM settings is not evaluated.

Relevance to Vision-Language Models

This paper addresses a core VLM challenge—attention grounding—by using biologically derived supervision (human gaze) rather than additional architectural components or larger data. For researchers tracking VLMs, the training-only gaze regularization paradigm is broadly applicable: it imposes no inference-time overhead and plugs into any transformer-based VLM, making it a lightweight technique for improving spatial grounding. The work connects to emerging lines of VLM alignment research (e.g., Voila-A, contrastive attention supervision) but uniquely targets temporally grounded, generative tasks in egocentric video. The demonstrated reduction in visual hallucinations through attention grounding suggests gaze regularization as a complementary approach to RLHF or contrastive decoding methods for hallucination mitigation.