Skip to content

HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space

🕒 Published (v1): 2026-06-30 07:21 UTC · Source: Arxiv · link

Why this paper was selected

Nassir Navab; surgical VLM pretraining in hyperbolic space; strong med contribution

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HyperVLP is the first surgical video-language pre-training framework that operates in hyperbolic space (Lorentz manifold) to explicitly model the hierarchical tree structure of surgical workflows (phases → steps → actions). It introduces a geometry-aware contrastive loss that down-weights structurally related false negatives and a cone-based entailment objective enforcing parent-child semantic containment, achieving consistent gains in zero-shot and few-shot phase recognition across procedures and institutions.

Problem

Existing surgical VLP methods (SurgVLP, HecVL, PeskaVLP) embed hierarchical surgical knowledge in Euclidean space, which lacks geometric inductive bias for cross-level semantic containment (actions belong to steps, steps compose phases). Standard contrastive objectives treat all non-matching video-text pairs as negatives, even when segments are semantically adjacent or temporally correlated within the same procedural context — creating structural false negatives that artificially separate clinically related nodes and distort the workflow's topological structure.

Method

  • Hyperbolic embedding: Visual and textual features are linearly projected to the tangent space at the origin \(o = [\sqrt{1/c}, 0]\) and mapped into the Lorentz manifold \(\mathcal{L}^n\) via the exponential map \(\exp_c^o(\cdot)\).
  • Geometry-aware contrastive loss (\(\mathcal{L}_{\text{GeoHCL}}\)): For each anchor \(i\), candidates are partitioned into structurally related samples \(\mathcal{S}(i)\) (contextual neighbors within the same procedure), matched positives \(\mathcal{P}(i)\), and negatives \(\mathcal{N}(i)\). An adaptive weight \(\alpha_{ik} = \sigma(-\sum_{k \in \mathcal{P}(i)} d_{ik} \cdot d_{ik} + \epsilon)\) suppresses structurally related samples that exhibit high similarity relative to true positives, preventing over-separation of semantically correlated clips.
  • Cone-based entailment loss: Adopts hyperbolic entailment cones in Lorentz space. A child embedding \(y\) is constrained to lie within the half-aperture \(\theta(x)\) of its parent \(x\); the loss \(\mathcal{L}_{\text{ent}}(y, x) = \max(0, \phi(y, x) - \eta\theta(x))\) penalizes the angular residual. Applied both inter-modal (textual concepts contain their corresponding video semantics: \(\mathcal{L}_{\text{inter}} = \mathcal{L}_{\text{ent}}(v_c, t_c) + \mathcal{L}_{\text{ent}}(V_c, T_c)\)) and intra-modal (phase contains step within each modality: \(\mathcal{L}_{\text{intra}} = \mathcal{L}_{\text{ent}}(t_c, T_c) + \mathcal{L}_{\text{ent}}(v_c, V_c)\)).
  • Two-stage training: Stage I optimizes only \(\mathcal{L}_{\text{GeoHCL}}\) for stable cross-modal alignment. Stage II freezes encoders, fine-tunes only hyperbolic mapping layers with \(\mathcal{L}_{\text{GeoHCL}} + \mathcal{L}_{\text{inter}} + \mathcal{L}_{\text{intra}}\).

Key Contributions

  • First hyperbolic surgical VLP framework that explicitly encodes the intrinsic tree structure of surgical workflows into a shared Lorentz manifold
  • Identifies structural false negatives in surgical contrastive learning and proposes a geometry-aware hyperbolic contrastive strategy with adaptive down-weighting of procedurally related samples
  • Cone-based hyperbolic entailment objective that enforces geometric containment between parent and child embeddings across both inter-modal and intra-modal relations
  • Progressive two-stage optimization that stabilizes hierarchical alignment by separating cross-modal alignment from entailment-based refinement

Results

  • Zero-shot phase recognition (Acc/F1): HyperVLP outperforms all baselines on all four benchmarks — Cholec80: 49.9/37.2 (PeskaVLP: 45.1/34.2), AutoLaparo: 42.9/32.9 (PeskaVLP: 26.5/23.6), Stras70: 52.7/41.1 (PeskaVLP: 46.7/28.6), Bern70: 50.0/30.3 (PeskaVLP: 45.7/22.6)
  • Linear probing (100% labels): Best across all datasets — Cholec80: 71.3/60.2, AutoLaparo: 65.3/51.1, Stras70: 75.6/62.3 (PeskaVLP: 71.4/59.5), Bern70: 76.9/45.1 (PeskaVLP: 71.5/37.4)
  • Few-shot (10% labels): Larger relative gains, indicating hyperbolic representations are more linearly separable — Stras70: 66.9/53.4 vs PeskaVLP 63.8/50.4; Bern70: 72.3/44.8 vs PeskaVLP 62.9/32.7
  • Ablation: Adding \(\mathcal{L}_{\text{GeoHCL}}\), \(\mathcal{L}_{\text{inter}}\), and \(\mathcal{L}_{\text{intra}}\) progressively improves performance; the full model (all three) yields the best results

Limitations

  • Evaluation is limited to phase recognition; action-level and step-level recognition are not reported as downstream tasks
  • Requires hierarchical annotations (phase-step-action structure) at pre-training time, limiting applicability to datasets without multi-level labels
  • Two-stage optimization adds training complexity and introduces a design choice about when to freeze encoders
  • Tested only on laparoscopic surgical domains (cholecystectomy, hysterectomy, gastric bypass); generalization to other surgical modalities (e.g., robotic, endoscopic) is unverified

Relevance to Vision-Language Models

This paper demonstrates that hyperbolic space provides a more appropriate geometric inductive bias than Euclidean space for VLMs operating on hierarchically structured data. The geometry-aware contrastive loss and cone-based entailment framework are general techniques applicable beyond surgery — any VLP task with tree-structured semantic relations (e.g., actions → tasks → goals in robotics, frames → scenes → episodes in video understanding) could benefit from this approach. It connects to the broader line of hyperbolic VLMs (MERU, CompoCLIP) but is the first to address the unique challenge of multi-level procedural hierarchy.