Skip to content

HTT-Net: Hierarchical Text-guided Transition Modeling for Surgical Video Phase Recognition

🕒 Published (v1): 2026-07-18 11:49 UTC · Source: Arxiv · link

Why this paper was selected

Text-guided surgical video phase recognition; strong medical VLM baseline with temporal structure

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HTT-Net introduces a hierarchical text-guided transition modeling framework for surgical video phase recognition by encoding structured surgical semantics (intra-phase, inter-phase transition, and fine-grained unit descriptions) into two novel modules that construct and calibrate phase-aware segment representations. The approach suppresses spurious prediction noise near phase boundaries and resolves visual ambiguity between similar phases without requiring dense frame-level vision-language fusion. On Cholec80, HTT-Net achieves 96.4% accuracy (relaxed), outperforming prior clip-based methods.

Problem

Visual-temporal models for surgical phase recognition suffer from three interrelated failures: (1) local visual ambiguity — tool motion, occlusion, bleeding, and camera movement cause spurious apparent phase changes; (2) transient prediction noise — frame-wise predictions around boundaries exhibit premature switching, delayed transitions, and short oscillatory segments; (3) underutilized procedural semantics — existing methods treat phase labels as flat discrete categories, ignoring rich structured knowledge about surgical goals, anatomical context, instrument usage, and procedural transition patterns.

Method

HTT-Net processes a \(T\)-frame clip \(V = \{I_t\}_{t=1}^T\) through a ConvNeXtV2-Tiny backbone to obtain frame features \(X \in \mathbb{R}^{T \times d}\) and preliminary phase logits \(P_v \in \mathbb{R}^{T \times C}\). A frozen PubMedBERT encoder pre-computes three text embedding banks: intra-phase descriptions \(H_P \in \mathbb{R}^{C \times d}\), inter-phase transition descriptions \(H_T \in \mathbb{R}^{N_T \times d}\), and fine-grained semantic units \(H_U \in \mathbb{R}^{N_U \times d}\).

TAS-Con (Transition-Aware Segment Construction): First stabilizes \(P_v\) by fusing a semantic phase prior \(P_s\) (computed by matching \(X\) against \(H_P\)): $\(P_f = (1 - \alpha)P_v + \alpha P_s\)$ Candidate change points are then scored by combining visual context contrast (comparing local left/right windows) with phase consistency cues. Each candidate is verified via cross-attention against \(H_T\); low-plausibility candidates (transient visual variation) are suppressed. Surviving candidates partition the clip into \(M\) segments; frame features are pooled into segment representations \(Z = [z_1, \ldots, z_M]\).

TAS-Calib (Transition-Aware Segment Calibration): Adapts the semantic banks to the current clip, then applies multi-head cross-attention between \(Z\) and each level \(k \in \{P, T, U\}\): $\(O_k = \text{MHCA}(Z, \hat{H}_k, \hat{H}_k)\)$ A hierarchical semantic fusion module integrates \(\{O_P, O_T, O_U\}\) with segment context into enhanced representations \(R = [r_1, \ldots, r_M]\), which are redistributed back to frames for final dense prediction.

Training uses a three-term loss: $\(\mathcal{L} = \mathcal{L}_\text{cls} + \lambda_\text{seg}\mathcal{L}_\text{seg} + \lambda_\text{bnd}\mathcal{L}_\text{bnd}\)$ where \(\mathcal{L}_\text{seg}\) uses pseudo labels from dominant frame labels per segment, and \(\mathcal{L}_\text{bnd}\) supervises candidate change scoring using ground-truth boundary frames.

Key Contributions

  • HTT-Net: end-to-end framework integrating a three-level hierarchical surgical semantic memory (intra-phase, inter-phase transition, fine-grained units) for phase recognition.
  • TAS-Con: transition-plausibility verification via cross-attention against inter-phase text embeddings, suppressing spurious visual changes and producing coherent phase-aware segments.
  • TAS-Calib: segment-level (not frame-level) cross-modal calibration with three complementary semantic levels, reducing computation while resolving inter-phase visual ambiguity.
  • Evaluation on Cholec80 and in-house LCRS-100 (23 phases, colorectal resection, higher complexity) with ablation studies isolating each component.

Results

Cholec80 (relaxed evaluation, accuracy ± std):

  • HTT-Net (clip-based): 96.4 ± 3.6% vs. B2Q-Net 95.8 ± 3.5%, Surgformer 93.4 ± 6.4%, FACT 94.6 ± 4.6%
  • HTT-Net-F (frame-based online): 95.9 ± 4.6% vs. B2Q-Net-F 95.3 ± 6.5%, Surgformer 93.4 ± 6.4%, SKiT 93.4 ± 5.2%

Cholec80 unrelaxed and LCRS-100 quantitative results are not fully present in the provided text excerpt; precision, recall, and Jaccard values are reported but cut off before disclosure.

Limitations

  • Hierarchical surgical textual descriptions appear to be manually authored or curated — the process for constructing and scaling them to new surgical procedure types is not described.
  • PubMedBERT text encoder is frozen throughout training; domain shift to new procedure types may require re-encoding or fine-tuning.
  • Evaluation is restricted to two laparoscopic datasets (cholecystectomy and colorectal resection); generalizability to open surgery, robotic surgery, or non-abdominal procedures is unverified.
  • LCRS-100 is an in-house dataset with no public release; reproducibility by external groups is limited.
  • Results table is truncated in the provided text, preventing full comparison of precision/recall/Jaccard scores.

Relevance to Vision-Language Models

HTT-Net demonstrates how frozen domain-specific text encoders (PubMedBERT) can inject structured procedural semantics into a video-temporal pipeline without end-to-end vision-language pretraining, a practical alternative to full VLM fine-tuning in data-scarce medical domains. The three-level semantic memory (identity, transition, fine-grained units) goes beyond simple class-name prompting, showing that richer text granularity — specifically inter-phase transition descriptions — is necessary for temporal boundary reasoning, which is a general insight for video-language models applied to structured activity recognition. The segment-level (rather than frame-level) cross-modal fusion strategy offers a scalability-accuracy trade-off relevant to long-video VLM architectures. This work connects to the growing literature on surgical VLMs (e.g., SurgicalVLP, CholecT50 grounding) and highlights the value of procedural knowledge encoding for temporally sensitive clinical video tasks.