Skip to content

From Human Attention to Diagnosis: Semantic Patch-Level Integration of Vision-Language Models in Medical Imaging

๐Ÿ•’ Published (v1): 2025-01-01 ยท Source: NeurIPS ยท Venue: NeurIPS 2025 ยท link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

LogitGaze-Med is a multimodal transformer framework for predicting radiologist scanpaths on chest X-rays by fusing domain-specific visual encoders (CheXNet), diagnostic text embeddings, and patch-level semantic priors extracted from a medical VLM (LLaVA-Med) via the logit-lens technique. It achieves 20โ€“30% gains over competitive baselines on scanpath similarity and improves downstream pathology classification AUROC by over 5% when predicted fixations augment training data.

Problem

Existing scanpath prediction methods rely on generic saliency features and free-viewing assumptions, failing to capture the goal-directed, semantically structured nature of radiological visual search. Clinical targets (e.g., "pleural effusion") lack well-defined spatial extent, gaze datasets are scarce, and prior approaches lack the domain-specific semantic grounding needed to predict clinically coherent fixation sequences.

Method

LogitGaze-Med processes chest X-rays through four stages:

  1. Logit-lens semantic extraction: The image is divided into a 24ร—24 patch grid (576 patches). Each patch's final hidden state from an instruction-tuned medical VLM (LLaVA-Med or LLaVA-Rad) is projected through the language model head; top-k vocabulary logits are decoded back into word vectors via the frozen embedding matrix and pooled into a per-patch semantic vector s_i โˆˆ R^d. This exposes latent clinical concepts (e.g., "consolidation," "effusion") at spatial locations.

  2. Feature preprocessing: Visual features come from a domain-specific CNN backbone (CheXNet/DenseNet-121, ResNet-COCO, CheSS, or PEAC) producing X_vis โˆˆ R^{Bร—Mร—2048}. Diagnostic labels are encoded via SentenceTransformer (stsb-roberta-base-v2) into task embeddings.

  3. Joint transformer encoding: Visual and semantic embeddings are linearly projected to shared dimension d, concatenated into F_joint โˆˆ R^{Bร—2Mร—d}, and passed through a 6-layer transformer encoder with 2D learnable positional encodings and broadcasted task embeddings. Loss is a weighted sum of token classification (stop token CE), spatial L1 regression (x,y coordinates), and temporal MSE regression (dwell duration).

  4. Scanpath regression: A transformer decoder with L learnable queries predicts, per fixation step, spatial means and log-variances for (x, y, ฯ„) via MLP heads; samples are drawn via the reparameterization trick to capture gaze variability.

Key Contributions

  • First application of logit-lens decoding to medical VLMs for patch-level, clinically grounded gaze prediction, with ablation studies validating each component.
  • Formulation of scanpath prediction as continuous coordinate + dwell-time regression (rather than discretized grid classification), enabling direct use as auxiliary training signal.
  • Demonstration that predicted fixations from LogitGaze-Med improve downstream 3-class pathology classification AUROC by >5% vs. image-only baseline.
  • Expert-validated synthetic scanpath generation for MIMIC-CXR (4.3/5.0 realism, 4.2/5.0 clinical relevance; radiologist achieved only 58% real-vs-synthetic discrimination accuracy).

Results

  • GazeSearch scanpath similarity (vs. GazeFormer, HAT, ChestSearch, LogitGaze):
  • ScanMatch w/o duration: 0.419 (LogitGaze-Med CheX) vs. 0.332 (best baseline, ChestSearch) โ€” +26% relative
  • ScanMatch w/ duration: 0.330 vs. 0.225 (LogitGaze) โ€” +48% relative
  • SED: 4.68 vs. 4.88 (ChestSearch) โ€” 4% reduction
  • STDE: 0.855 vs. 0.810 (LogitGaze)
  • MultiMatch (CheX): Vector 0.938, Direction 0.651, Length 0.948, Position 0.823, Duration 0.740 โ€” best across all five dimensions vs. all baselines.
  • Pathology classification AUROC (MIMIC-CXR, 3-class):
  • Baseline (image only): 0.82; Temporal pipeline: 0.90; U-Net: 0.91 (all best among gaze sources)
  • vs. LogitGaze synthetic gaze: Temporal 0.87โ†’0.90, U-Net 0.90โ†’0.91
  • Ablation: Image-only ScanMatch 0.148; +text 0.280; full model 0.419. Shuffled semantic alignment causes >40% ScanMatch drop.
  • Inference overhead: 85 ms vs. 70 ms per sample (logit-lens extraction cost).

Limitations

  • Logit-lens is sensitive to noisy activations from non-content tokens (punctuation, function words), causing potential fixation errors from unstable noun activations.
  • Training on synthetic scanpaths risks overfitting to model-generated gaze distributions.
  • Specialization for chest X-rays hurts cross-domain generalization: on COCO-Search18, MultiMatch drops to 0.749 vs. 0.862 for LogitGaze.
  • Validation limited to a single board-certified radiologist evaluating 100 images; multi-institution and multi-expertise-level validation is absent.
  • PEAC/CheSS encoders yield only marginal gains over CheXNet, suggesting diminishing returns from encoder upgrades alone.

Relevance to Vision-Language Models

This paper demonstrates a novel inference-time use of medical VLMs (LLaVA-Med, LLaVA-Rad): rather than generating text outputs, the logit-lens probes the model's internal representations to extract patch-level clinical semantics without additional fine-tuning, effectively repurposing the VLM as a grounded semantic feature extractor. It connects to the broader VLM interpretability literature (logit-lens, layer-wise concept emergence) and extends it to a spatially structured prediction task, showing that intermediate activations of instruction-tuned VLMs encode localized, clinically meaningful concepts. For VLM researchers, it illustrates how to bridge VLM representations with human visual attention and use that bridge to improve both scanpath modeling and downstream discriminative tasks.