VEGAS: Human-Aligned Video Caption Evaluation via Gaze

🕒 Published (v1): 2026-07-09 13:46 UTC · Source: Arxiv · link

Why this paper was selected

Gaze-grounded VLM caption eval; training-free metric; novel human-attention alignment signal

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VEGAS (Video caption Evaluation via GAze Score) is a training-free, cross-modal metric that uses test-time viewer gaze to select video captions aligned with individual visual attention, without retraining the underlying VLM. It approximates mutual information between captions and gaze via pretrained VLM token likelihoods, then applies rejection sampling over candidate captions. On egocentric video, VEGAS-selected captions significantly outperform naive VLM captions in semantic alignment and downstream retrieval.

Problem

VLMs produce video captions trained on crowd-sourced, aggregated annotations that reflect globally visible content rather than any individual viewer's attentional focus. In egocentric or retrieval-centric settings, this global description misaligns with what specific users attend to—making the caption a poor index for personalized search or action recognition.

Method

VEGAS defines caption quality as the pointwise conditional mutual information between caption tokens \(t\) and the non-attended visual regions \(\bar{G} = V \setminus G\), conditioned on the attended gaze regions \(G\):

\[\text{VEGAS}(c, G) \triangleq I(t;\bar{G} \mid G) = \log \frac{P_\theta(t \mid V)}{P_\theta(t \mid G)} \quad \text{(lower is better)}\]

Because a frozen VLM factorizes likelihoods autoregressively, this decomposes token-by-token:

\[\text{VEGAS}(c, G) = \sum_{\ell=1}^{L} \log \frac{P_\theta(t_\ell \mid t_{<\ell}, V)}{P_\theta(t_\ell \mid t_{<\ell}, G)}\]

A low score means the caption is well-predicted from attended regions alone—non-attended content adds little lift. At inference, \(N\) candidates are drawn from one or more VLMs at multiple temperatures; the candidate minimizing VEGAS is selected:

\[c^* = \arg\min_{c_1,\ldots,c_N} \text{VEGAS}(c_i, G)\]

No gradient updates or fine-tuning are required; gaze is used only to spatiotemporally mask frame regions before VLM forward passes.

Key Contributions

• VEGAS metric: reference-free, training-free, cross-modal caption evaluator grounded in pointwise conditional mutual information between gaze and language.
• Gaze-conditioned rejection sampling: inference-time framework for selecting personalized captions from any VLM without modifying model weights.
• Multimodal benchmark dataset: 632 visual samples (332 AEA egocentric clips + 300 SlideVQA slides) paired with 2,981 human captions and synchronized gaze signals across two complementary domains.
• Token-level interpretability: per-token VEGAS scores identify which semantic concepts are or are not supported by the viewer's gaze.
• VLM-agnostic design: applicable to any model exposing token-level conditional likelihoods, including closed-source APIs.

Results

• AEA semantic alignment: VEGAS-selected captions achieve +0.0856 mean SBERT similarity over naive Gemini captions (Wilcoxon signed-rank test, statistically significant).
• AEA retrieval (caption-to-video mAP): +1.14% at rank 1, +2.48% at rank 5, +2.46% at rank 10 over conventional VLM-generated indexing.
• SlideVQA: SBERT improvement is +0.0256 and non-significant, indicating VEGAS is domain-sensitive.
• Human caption diversity (AEA): mean pairwise human-human embedding distance = 1.53 vs. Gemini–centroid distance = 0.48, motivating personalized approaches.
• Scorer robustness: downstream SBERT and retrieval metrics remain stable across top-\(k\) token probability thresholds and an alternative Qwen2.5-VL-32B-Instruct scorer.

Limitations

• Gaze is an imperfect and indirect proxy for cognitive attention; internal perceptual reasoning is inaccessible.
• VEGAS is most effective when gaze disambiguates concrete object referents; it is less diagnostic for stimuli that admit many valid high-level summaries (demonstrated by non-significant SlideVQA results).
• Requires gaze hardware (Project Aria glasses for egocentric; webcam eye-tracking for slides) at inference time, limiting deployment to gaze-equipped settings.
• VEGAS scoring with Gemma-4-31B-IT locally imposes non-trivial compute overhead per candidate caption.
• Dataset scale is modest (332 AEA clips, 300 slides); generalization to broader video domains is not demonstrated.

Relevance to Vision-Language Models

VEGAS directly exploits the internal token-level conditional likelihoods of frozen VLMs as a scoring function, requiring no architectural change—making it immediately applicable to any VLM that exposes these probabilities, including API-served models like GPT and Gemini. It highlights a systematic gap in how VLMs aggregate crowd-sourced training signal, producing descriptions that cluster near a semantic consensus rather than individual attentional foci—a finding with direct implications for personalized VLM deployment. The information-theoretic framing extends a growing line of cross-modal, reference-free VLM evaluation metrics and offers a concrete alternative to BLEU/METEOR/CIDEr for visually grounded caption quality. The rejection sampling approach is a lightweight inference-time alignment technique complementary to RLHF or DPO fine-tuning, broadening the toolkit for steering VLM outputs without gradient updates.