Skip to content

Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HiVE-MIL is a hierarchical vision-language multiple instance learning (MIL) framework for few-shot classification of gigapixel whole slide images (WSIs). It constructs a unified hierarchical heterogeneous graph that explicitly models both cross-scale (5×→20×) interactions within each modality and intra-scale visual–textual alignment. On three TCGA cancer datasets (lung, breast, kidney) it consistently outperforms prior VLM-based MIL methods in 4-, 8-, and 16-shot settings.

Problem

Existing multi-scale VLM-based MIL methods for few-shot WSI classification (FSWC) suffer from two compounding failures: (1) they process visual and textual features independently at each magnification scale and merge them only at prediction time, discarding hierarchical semantic progression within each modality; (2) they perform inadequate intra-scale cross-modal alignment, relying on simple summation/averaging instead of structured patch–text interaction. Together these gaps limit data-efficient learning when labeled WSIs are scarce.

Method

HiVE-MIL builds a Hierarchical Heterogeneous Graph (HHG) over four node types — low-scale image (5×), high-scale image (20×), low-scale text, high-scale text — with two complementary edge sets:

  • Hierarchical edges link parent (coarse 5×) to child (fine 20×) nodes separately for each modality, using spatial coordinate mapping for images and LLM-generated parent–child morphological descriptions (via GPT-4o) for text.
  • Intra-scale edges connect visual and textual nodes at the same magnification, but only for semantically valid pairs retained by Text-Guided Dynamic Filtering (TGDF): a two-stage, text-wise soft-thresholding module (threshold = µ + α·σ) that first filters low-scale patch–text pairs, then propagates that mask to prune high-scale pairs.

Graph learning uses a Hierarchical Heterogeneous GNN (HHGNN): - Intra-scale aggregation via relation-specific GraphSAGE. - Cross-scale aggregation via Modality-Scale Attention (MSA), an attention mechanism conditioned on both modality type and scale direction (scale embeddings + relation-specific QKV projections).

Training combines cross-entropy loss with a Hierarchical Text Contrastive Loss (HTCL) that aligns low- and high-scale class-level text embeddings using cosine similarity with class-wise positives/negatives.

Key Contributions

  • HHG construction: unified graph encoding parent–child (hierarchical) and same-scale cross-modal (intra-scale) relationships across 5× and 20× visual/textual nodes.
  • TGDF module: two-stage text-wise soft thresholding that dynamically removes weakly correlated patch–text pairs, yielding non-fixed intra-scale edges updated during training as learnable text tokens evolve.
  • MSA mechanism: attention over hierarchical edges that conditions on both modality type and scale, outperforming scale-agnostic or modality-agnostic variants by 1–3%.
  • HTCL: class-wise hierarchical contrastive loss enforcing semantic consistency between parent (low-scale) and child (high-scale) text embeddings.
  • Comprehensive evaluation across 3 TCGA datasets × 3 pathology VLMs (PLIP, QuiltNet, CONCH) × 4-/8-/16-shot settings.

Results

  • 16-shot (main): HiVE-MIL outperforms the second-best baseline on all 9 dataset×VLM combinations:
  • TCGA-BRCA: up to +4.37% ACC, +4.22% AUC, +4.14% macro F1 (QuiltNet).
  • TCGA-NSCLC: up to +3.27% ACC, +3.28% AUC, +3.26% macro F1 (PLIP).
  • TCGA-RCC: up to +1.07% ACC, +0.85% AUC, +1.64% macro F1 (PLIP).
  • 4-/8-shot: gains of up to +6.81% (4-shot NSCLC, QuiltNet) and +8.57% (8-shot NSCLC, QuiltNet) AUC over the best baseline.
  • Ablation: removing HTCL drops macro F1 by 2.04% on BRCA; removing HHG further drops 1.64% on RCC; removing TGDF degrades all datasets consistently.
  • MSA vs. alternatives: MSA outperforms Scale-Aware Attention, Modality-Aware Attention, generic Attention, and GraphSAGE by 1–3% macro F1.
  • Hit Ratio@2 (hierarchical text alignment metric) is highest for bidirectional HiVE-MIL and positively correlated with macro F1 across all settings.

Limitations

  • TGDF thresholds (µ + α·σ) are non-learnable and computed per-WSI, which may reduce generalization across datasets or different backbone encoders.
  • The Hit Ratio metric assumes correctness of LLM-generated parent–child text hierarchies; no expert human validation of these hierarchies is reported.
  • Moderate computational overhead relative to simpler MIL baselines (FLOPs and GPU memory quantified in appendix but not assessed at clinical deployment scale).
  • Only evaluated on TCGA data; performance on out-of-distribution cohorts or staining variations is not tested.

Relevance to Vision-Language Models

HiVE-MIL advances the integration of domain-adapted VLMs (PLIP, QuiltNet, CONCH) into structured MIL frameworks by replacing naive feature concatenation with graph-based hierarchical alignment, a direction increasingly important as VLMs are applied to specialized, high-resolution domains where natural-image pretraining is insufficient. The TGDF and HTCL components address a recurring challenge in VLM adaptation: ensuring that learned text prompts remain semantically grounded to the correct visual regions rather than drifting to spurious correlations. The hierarchical text prompt construction via LLM (GPT-4o) and learnable CoOp-style context tokens represents a scalable strategy for injecting domain structure into VLM text encoders without full fine-tuning. This work is directly relevant to researchers studying multi-scale VLM grounding, weakly-supervised multimodal learning, and prompt engineering for specialized vision domains.