LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition¶

🕒 Published (v1): 2026-07-22 08:20 UTC · Source: Arxiv · link

Why this paper was selected

Medical VLM fine-tuning for surgical instrument-tissue interaction; Intel Labs; reproducible

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LAViFiT is an end-to-end fine-tuning framework that adapts pretrained vision encoders and VLMs to surgical instrument-tissue interaction recognition by jointly training an inverse dynamics model (IDM) and a forward world model (FWM) to bias the encoder toward action-relevant spatial regions. A patch-level SIG regularizer prevents local feature collapse during training, requiring no bounding-box or pseudo-label supervision. The approach yields improved triplet recognition and stronger spatial grounding in instrument-tissue interaction regions across multiple encoder backbones.

Problem¶

Standard strategies for adapting pretrained VLMs to surgical action-triplet recognition ($\langle$instrument, verb, target$\rangle$) fail in two complementary ways: (1) freezing the vision encoder propagates noise and spatially imprecise features from weakly-aligned surgical pretraining data; (2) full fine-tuning optimizes global semantic alignment without guaranteeing that the encoder localizes features to the actual instrument-tissue interaction region, leaving models susceptible to shortcut features (background, irrelevant motion). Prior solutions requiring instrument trajectories, segmentation masks, or region pseudo-labels introduce preprocessing overhead and tend to overemphasize instrument regions while neglecting the target tissue.

Method¶

LAViFiT jointly trains five components — vision encoder $E_\phi$, IDM $I_\theta$, FWM $F_\psi$, clip aggregator $A_\omega$, and text encoder $G_\chi$ (EmbeddingGemma) — under a combined objective.

Delta-IDM. Rather than concatenating two state tokens, the IDM operates on their difference $\delta_t = s_{t+1} - s_t$, passes it through a causal Transformer, and projects to a low-dimensional latent action $a_t \in \mathbb{R}^{d_a}$ ($d_a \ll d_s = 128$). Operating on differences discourages the model from bypassing the bottleneck via endpoint-state shortcuts.

FWM. A DiT-style Transformer with adaLN-zero conditioning predicts $\hat{s}_{t+1} = F_\psi(s_t, a_t)$ against a stop-gradient target $\text{sg}(s_{t+1})$: $$\mathcal{L}_\text{pred} = \frac{1}{T}\sum_{t=0}^{T-1} \|F_\psi(s_t, a_t) - \text{sg}(s_{t+1})\|_2^2$$ The stop-gradient prevents representational collapse of the encoder states.

Aggregator. A bidirectional Transformer fuses the sequence $[\text{AGG}, s_0, \ldots, s_T, a_0, \ldots, a_{T-1}]$; the [AGG] token is projected to text embedding space and matched against triplet text embeddings via cosine similarity logits.

SIG Regularizer. Both state and action representations are regularized toward $\mathcal{N}(0, I)$ using the sliced Cramér-Wold Epps-Pulley statistic over $M=1024$ random projections on the unit sphere. A separate patch-level variant applies the same regularizer to a random subset of 32 patch tokens per frame ($M=256$ projections), preventing local feature collapse that the state-level term alone cannot catch.

Total loss: $$\mathcal{L} = \mathcal{L}_\text{cls} + \lambda_\text{pred}\mathcal{L}_\text{pred} + \lambda_s \mathcal{L}_\text{SIG} + \lambda_p \mathcal{L}^\text{patch}_\text{SIG}$$ with $\mathcal{L}_\text{cls}$ a multi-label BCE over triplets and component (instrument/verb/target) heads.

Key Contributions¶

End-to-end latent-action fine-tuning framework (IDM + FWM) that reshapes pretrained vision encoders toward instrument-tissue interaction regions without region-level labels or motion preprocessing.
Delta-IDM operating on consecutive-state differences to discourage transition collapse and endpoint shortcuts.
Patch-level SIG regularizer that prevents local feature collapse at tractable cost by subsampling 32 patches per frame.
Systematic encoder-behavior analysis (Calinski-Harabasz patch cluster quality, feature transition localization) demonstrating improved spatial coherence and grounding in instrument-tissue regions.
Evaluation spanning seven encoders from two pretraining regimes (general-domain: CLIP, DINOv2, V-JEPA; surgical-domain: EndoViT, SurgeNet, SurgVLP, HecVLP, PeskaVLP, LEMON) on CholecT50 and ProstaTD.

Results¶

The quantitative results table is not present in the provided text (paper was truncated before §4.1 numbers). The following qualitative claims are stated: - Consistent improvements in triplet mAP (mAP$_\text{IVT}$) and per-component APs over standard fine-tuning baselines across all three instantiated encoders (ViT-L/16, DINOv2-L/14, SurgeNet-L/14) on CholecT50 and ProstaTD. - Ablation shows that the states-only model (CLIP4Clip-style, no IDM/FWM) underperforms LAViFiT, isolating the contribution of the latent-action module. - Patch-level SIGReg further improves over LAViFiT without it. - Encoder-behavior analyses show stronger Calinski-Harabasz patch cluster quality and better localization of frame-to-frame feature transitions within instrument-tissue interaction masks.

Limitations¶

Quantitative results were not available in the provided text excerpt.
The method requires multi-frame clips (8 frames per labeled frame); single-frame image encoders cannot leverage the IDM/FWM module without temporal context, requiring architectural adaptation.
The interaction region mask for evaluation relies on SASVI propagation of CholecSeg8k annotations, which covers only 8 semantic classes (2 instruments, 6 tissues); generalization to other surgical procedures or instrument sets is untested.
Theoretical guarantees for the Delta-IDM bottleneck (cited from [47]) note that structured nuisance variation may also be retained alongside action-induced transitions.
The framework is evaluated on laparoscopic cholecystectomy and robot-assisted prostatectomy only; cross-procedure generalization is not assessed.

Relevance to Vision-Language Models¶

LAViFiT directly addresses a practical bottleneck in VLM adaptation: how to fine-tune the vision encoder so that learned patch features are spatially grounded in semantically relevant regions, not just globally aligned to text. The approach demonstrates that latent-action objectives — originally developed for robot control — can serve as a self-supervised spatial regularizer that complements contrastive image-text alignment, an insight transferable to any VLM fine-tuning scenario where localized feature quality matters (e.g., medical VLMs, visual grounding, compositional recognition). The use of EmbeddingGemma as a text encoder and cosine-similarity logits for multi-label triplet classification mirrors the CLIP-style paradigm, making the method directly applicable to the broader VLM fine-tuning literature. The patch-level SIG regularizer is a lightweight, annotation-free alternative to region-contrastive objectives (e.g., RegionCLIP, GroupViT) for improving local feature quality in pretrained ViT backbones.