Skip to content

ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ExGra-Med introduces a multi-graph alignment framework for medical multimodal LLMs that jointly aligns image, instruction-following, and GPT-4-extended caption embeddings via a combinatorial graph-matching objective. It addresses the data-hunger of autoregressive pretraining by enforcing structural triplet consistency in latent space. With only 10% of LLaVA-Med's pretraining data, it matches or exceeds full-data performance on medical VQA benchmarks.

Problem

Standard medical MLLMs (e.g., LLaVA-Med, BioMedGPT) rely on autoregressive objectives that are highly data-dependent: reducing pretraining instruction-following data from 100% to 10% causes LLaVA-Med to drop from 72.64% to 52.39% on VQA-RAD. Scaling model size or data volume is the default fix, but weak vision-language alignment persists without sufficient instruction-tuning data. No prior work addresses the fundamental alignment weakness of autoregressive pretraining in the medical domain.

Method

ExGra-Med constructs three modality-specific graphs per training batch: \(G_v\) (visual patch embeddings from CLIP-ViT-L), \(G_a\) (LLM token embeddings of original instruction answers), and \(G_{ae}\) (LLM embeddings of GPT-4-extended answers). Extended answers are generated offline by prompting GPT-4 to add conceptual elaboration to each original answer without changing intent.

A barycenter graph \(G_{br}\) is defined with nodes averaging features across the three modalities. Multi-graph alignment is then reduced to \(K=3\) pairwise second-order graph alignment (SGA) problems between each modality graph and \(G_{br}\), formulated as quadratic assignment problems minimizing joint vertex+edge affinity. Because SGA is NP-hard and non-differentiable, the authors use Lagrange decomposition heuristic solvers for forward passes and Implicit Maximum Likelihood Estimation (IMLE) with Gumbel noise perturbations for black-box gradient estimation in backpropagation. A 2-layer GCN performs message passing on each graph to enrich node representations. The multi-graph alignment loss (Hamming between predicted and true barycenter assignments) is jointly optimized with the standard autoregressive language modeling loss, weighted by coefficient \(\alpha\). The LLM (LLaMA-7B) and projector are trained end-to-end; the vision encoder (CLIP-ViT-L/14) is frozen.

Key Contributions

  • Empirical demonstration that autoregressive pretraining is severely data-hungry in the medical domain, with a controlled ablation across 10%/40%/70%/100% data regimes.
  • Multi-graph alignment framework that enforces triplet-level structural consistency among image, instruction, and extended-caption embeddings via a shared barycenter graph, reducing \(K\)-graph alignment to \(K\) pairwise problems.
  • Scalable end-to-end training for LLaMA-7B using IMLE-based black-box gradient estimation through a combinatorial (non-differentiable) solver.
  • Theoretical results: proof that the SGA-derived distance \(d_{\text{SGA}}\) is a valid metric on structured graph space \(S(F)\) and that \(S(F)\) is geodesic under this metric.
  • ExGra-Med + DCI variant integrating multi-scale visual features (local + global) for improved pathology analysis.

Results

  • VQA-RAD (10% pretraining): ExGra-Med 72.52% vs. LLaVA-Med (10%) 52.39% — +20.13% gain; matches LLaVA-Med (100%) 72.64%.
  • VQA-RAD (100% pretraining): ExGra-Med 74.91% avg vs. LLaVA-Med 72.64%; ExGra-Med+DCI 75.25%.
  • SLAKE (100%): ExGra-Med 85.46% avg vs. LLaVA-Med 83.43%; ExGra-Med+DCI 85.23%.
  • PathVQA (100%): ExGra-Med+DCI 64.82% avg vs. LLaVA-Med 64.06%.
  • vs. Med-Dr (40B): ExGra-Med (7B) surpasses Med-Dr (58.2% on VQA-RAD) at all three VQA datasets despite 5.7× fewer parameters.
  • Zero-shot microscopy classification: ExGra-Med surpasses RadFM by 8.2%.
  • Training overhead: +0.5 hours over LLaVA-Med on 4×A100-80GB (total ~14 hours vs. ~13.5 hours).
  • LoRA finetuning (VQA-RAD): ExGra-Med 71.48% vs. LLaVA-Med 68.53%.
  • Extended captions alone (LLaVA-Med autoregressive, 10%): 69.41% — better than baseline but still below ExGra-Med 72.52%, confirming graph alignment adds value beyond data augmentation.

Limitations

  • Evaluation is primarily on the LLaVA-Med architecture; generalization to Flamingo-style models is unverified.
  • Extended captions are generated by GPT-4, introducing potential hallucination; a user study found mostly high quality but some low-rated outputs.
  • LoRA finetuning performance lags full finetuning; bridging the gap likely requires much larger pretraining data (e.g., 21M MedTrinity samples).
  • Graph construction uses simple average pooling and k-NN edges, which may not capture all intra-modal structure.
  • Extended captions are discarded at finetuning time, so any conceptual richness they encode must be fully captured during pretraining.
  • No evaluation on medical visual chain-of-thought or multi-step reasoning tasks.

Relevance to Vision-Language Models

ExGra-Med directly challenges the dominant VLM pretraining paradigm — scaling autoregressive instruction-following data — by showing that structural alignment losses can compensate for data scarcity. The multi-graph alignment formulation is a principled generalization of contrastive learning (e.g., CLIP/InfoNCE) to three or more modalities with relational (edge-level) constraints, relevant to any domain where pairwise contrastive objectives are insufficient. The IMLE-based black-box gradient trick enables this non-differentiable alignment to be plugged into large LLM training loops, which is a broadly applicable technique for discrete structure learning in VLMs. The result that graph-structured alignment outperforms multi-modal contrastive baselines (InfoNCE, SigLIP, IMAGEBIND, PAC-S) across all data regimes has direct implications for efficient VLM pretraining beyond the medical domain.