Skip to content

DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction

🕒 Published (v1): 2026-07-02 12:25 UTC · Source: Arxiv · link

Why this paper was selected

Autoregressive scanpath modeling via VLM token prediction; novel attention modeling approach

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DeepGaze3.5-VL reframes human scanpath prediction as autoregressive token generation by fine-tuning a Large Vision-Language Model (InternVL3.5-8B with LoRA) to predict discrete gaze coordinate tokens. This formulation directly aligns scanpath modeling with the VLM's native next-token-prediction objective, enabling exact Information Gain (IG) computation without heuristic surrogates. The model achieves 2.18 bits IG on MIT1003, a 46% improvement over a matched-backbone specialist baseline.

Problem

Existing scanpath models (e.g., DeepGaze III, TPPGaze, HAT) rely on specialized architectures with hand-crafted geometric and temporal priors (Lévy exploration, inhibitory tagging, saccadic bias modules), which are rigid and require bespoke extensions for task conditioning or observer personalization. Continuous generative scanpath models cannot natively compute exact spatial probability densities, forcing reliance on heuristic sequence-alignment metrics (ScanMatch, MultiMatch) that do not directly optimize Information Gain.

Method

Spatial coordinates are encoded as zero-padded two-digit integers in \([00, 99]\), representing percentage positions along each image dimension. This guarantees every fixation contributes exactly four coordinate tokens \((x_1, x_2, y_1, y_2)\), ensuring uniform tokenization irrespective of spatial position. The joint scanpath log-likelihood factorizes as:

\[\log P(\mathcal{S} \mid I, \mathcal{C}) = \sum_{i=1}^{n} \log P\bigl((x_i, y_i) \mid I, \mathcal{C}, (x_1,y_1),\ldots,(x_{i-1},y_{i-1})\bigr)\]

InternVL3.5-8B is parameter-efficiently fine-tuned via LoRA on ground-truth scanpath sequences. Conditioning factors \(\mathcal{C}\) (observer identity, task description, fixation durations) are injected as natural-language prefixes with no architectural changes.

Exact prefix-tree evaluation: The joint is decomposed as \(P(x,y) = P(x_1) \cdot P(x_2|x_1) \cdot P(y_1|x_1,x_2) \cdot P(y_2|x_1,x_2,y_1)\), evaluated in 4 KV-cached phases requiring \(1+10+100+1{,}000 = 1{,}111\) forward passes to obtain the complete \(100 \times 100\) probability map.

Rapid IG evaluation: Conditioning on the ground-truth fixation at each step requires exactly 4 forward passes per fixation to extract the exact scalar \(P(x^\text{GT}, y^\text{GT})\). Raw logits are renormalized over the valid digit set via log-sum-exp to eliminate probability leakage to non-digit tokens.

Duration conditioning augments the coordinate tuple with a discretized duration token \((x, y) \to (x, y, d)\), training the model to leverage pre-saccadic fixation durations for improved spatial prediction.

Key Contributions

  • Reformulates scanpath prediction as autoregressive discrete token prediction over VLMs, achieving state-of-the-art IG without any task-specific architectural priors.
  • Derives that Information Gain is a direct manifestation of the next-token-prediction objective, enabling exact, efficient IG computation via a 4-phase prefix tree (4 forward passes for IG; 1,111 for full spatial densities).
  • Demonstrates that general multimodal reasoning ability (MMMU score) correlates with scanpath prediction quality at \(\rho = 0.93\) (min LOO \(\rho = 0.90\)) across four VLM families and nine model scales.
  • Shows flexible conditioning — observer identity, task type, fixation duration — via natural-language prompts, without bespoke modules.
  • Enables counterfactual in-silico interventions on fixation duration, recovering known oculomotor phenomena (e.g., broad predictive distributions for long pre-saccadic fixations) purely from data.

Results

  • MIT1003: 2.18 bits IG (Combined); vs. matched-backbone DeepGaze III–6B: 1.49 bits (+46%); vs. TPPGaze: 1.18 bits; AUC 0.94.
  • COCO-FreeView: 2.19 IG / 0.94 AUC (Combined); DeepGaze III–6B: 1.69.
  • CAT2000: 2.07 IG / 0.95 AUC; DeepGaze III–6B: 1.60.
  • DAEMONS: 2.75 IG / 0.94 AUC; DeepGaze III–6B: 2.32.
  • FiGrIm: 1.93 IG / 0.94 AUC; DeepGaze III–6B: 1.59.
  • LODO (OOD): 1.94 IG on MIT1003 — matches or exceeds DeepGaze III–6B in-distribution (1.49) without seeing the target dataset.
  • InternVL 3.5–4B achieves ≥2 bits IG, matching or exceeding models 3× its parameter count.
  • SmolVLM2 plateaus at ~0.85 bits across 256M–2.2B scales, demonstrating architecture quality dominates raw scale.
  • DeepGaze III retrained with the larger InternViT-6B (vs. InternViT-300M used in the VLM) achieves only 1.49 vs. 1.45 bits, confirming the LLM branch — not the vision encoder — drives improvement.

Limitations

  • Spatial discretization to a \(100 \times 100\) grid caps resolution; fine-grained fixation positions are lost.
  • Full spatial density evaluation requires 1,111 forward passes per fixation, which remains expensive for large-scale deployment with 8B-parameter models.
  • SmolVLM2 architecture shows that not all VLM families benefit equally — performance is sensitive to the quality of pretrained multimodal representations, not just model size.
  • Evaluation on task-driven datasets (visual search) and duration conditioning results are present but the text is truncated, limiting full assessment of those contributions.
  • Interventional analysis is purely in-silico; causal claims about oculomotor mechanisms require additional validation against controlled behavioral experiments.

Relevance to Vision-Language Models

This paper provides direct empirical evidence that the sequential and semantic representations learned by LVLMs during pretraining transfer to low-level psychophysical prediction tasks — specifically, modeling where humans look over time. The Spearman \(\rho = 0.93\) correlation between MMMU score and scanpath IG establishes a quantitative link between general multimodal reasoning capability and visual attention modeling, suggesting that VLM pretraining implicitly encodes human-like attentional priors. The matched-backbone ablation isolates the LLM branch as the source of improvement (not the vision encoder), which has implications for how VLMs process and integrate visual context. The work also demonstrates a practical benefit of the autoregressive formulation: exact likelihood computation aligns model training directly with the evaluation metric, a desirable property that is generally unavailable in diffusion-based or point-process generative VLM applications.