Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation¶

🕒 Published (v1): 2026-07-09 12:21 UTC · Source: Arxiv · link

Why this paper was selected

Attribute-driven compositional referring segmentation for endoscopy; medical VLM with fine-grained control

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces ReferEndoscopy, a large-scale benchmark of 65,964 endoscopic images with 1.45M image–mask–instruction triplets, and proposes AR-ERIS, a framework for open-vocabulary referring image segmentation in endoscopy. AR-ERIS combines frequency-aware feature fusion (FFT decomposition into high/low components) with a test-time attribute retrieval module (ARM) that generates compositional text prompts for seen and novel classes without retraining.

Problem¶

Referring Image Segmentation (RIS) in endoscopy lacks large-scale, text-grounded benchmarks, and existing vision–language models fail to capture fine-grained anatomical/instrumental attributes (color, shape, texture, spatial location) in endoscopic imagery. Domain-specific challenges—occlusion, low lighting, blurred boundaries, class imbalance, and long-tail distributions—further limit generalization of natural-image RIS methods to surgical settings.

Method¶

ReferEndoscopy benchmark: Aggregates 10 endoscopic/laparoscopic datasets (AutoLaparo, CholecSeg8k, EndoVis-17/18, SISVSE, etc.) into 59 categories with an automated attribute annotation pipeline using OpenCV to extract color (HSV/RGB histograms), size (pixel-count area $A = \sum_{(x,y)\in\text{mask}} 1$), spatial location (centroid $x_c, y_c$), texture, and reference-object proximity. Instructions are templated at four granularity levels (naive/simple/medium/hard: 0/1/3/5 attributes).

AR-ERIS / FAR-Net: Built on CRIS (CLIPSeg backbone). For each input $I \in \mathbb{R}^{B\times3\times H\times W}$, a 2D FFT decomposes each channel into low-frequency ($M_\text{low}$, radius $r_\text{low}$) and high-frequency ($M_\text{high} = 1 - M_\text{low}$) components via IFFT reconstruction: $I_\text{low}$, $I_\text{high}$. These are concatenated with the original RGB: $I_\text{combined} \in \mathbb{R}^{B\times9\times H\times W}$, then fused via a mixture-of-experts network. Cross-modal attention aligns fused visual features with CLIP text embeddings. Training loss: $$\mathcal{L}_\text{total} = \mathcal{L}_\text{BCE} + \lambda_\text{freq}\cdot\mathcal{L}_\text{freq} + \lambda_\text{Dice}\cdot\mathcal{L}_\text{Dice}$$ with $\lambda_\text{freq}=0.15$, $\lambda_\text{Dice}=0.3$, where $\mathcal{L}_\text{freq} = \|I_\text{high} - M_\text{mask}\|_2^2$ enforces edge consistency.

ARM (Attribute Retrieval Module): At inference, a class-attribute memory bank built during pretraining supports top-$k$ retrieval. For novel classes not seen during training, ARM embeds the class name via CLIP and retrieves the most similar pretraining class's attributes via cosine similarity, then constructs an instruction prompt dynamically.

Key Contributions¶

ReferEndoscopy benchmark: 65,964 images, 242,055 masks, 1,452,330 triplets across 59 classes from 10 diverse endoscopic datasets, with automatically generated compositional instructions at four attribute-complexity levels.
FAR-Net: Frequency-aware feature fusion that explicitly separates boundary (high-frequency) and texture (low-frequency) information via FFT, enabling adaptive multi-frequency visual encoding for endoscopic domain.
ARM module: Test-time attribute retrieval enabling open-vocabulary segmentation of novel classes without retraining, by matching new class names to pretraining class-attribute co-occurrence memory via CLIP similarity.
State-of-the-art performance on ReferEndoscopy with demonstrated generalization to unseen classes and cross-domain endoscopic data.

Results¶

vs. LAVT (best prior baseline): AR-ERIS (medium + $\mathcal{L}_\text{Dice}$) achieves 74.64% mIoU / 85.48% DICE overall vs. LAVT's 38.75% mIoU / 55.86% DICE — roughly 2× improvement.
vs. EVF-SAM: 74.64% vs. 10.47% mIoU overall.
vs. GroundedSAM: 74.64% vs. 9.84% mIoU overall.
On the "hard" (5-attribute) subset, AR-ERIS (medium + $\mathcal{L}_\text{Dice}$) achieves 75.70% mIoU vs. LAVT's 35.58%.
Adding $\mathcal{L}_\text{Dice}$ consistently improves over the hard-pretraining variant across all subsets (e.g., overall 74.64% vs. 73.45%).

Limitations¶

Automated attribute annotation relies on OpenCV heuristics (color histograms, centroid geometry), which may introduce noise for ambiguous or overlapping regions.
Benchmark covers only endoscopic/laparoscopic procedures; generalization to other medical imaging modalities (CT, MRI) is untested.
Rare classes (<500 samples, ~0.2% of annotations) remain underrepresented; long-tail performance is not separately reported in the provided results.
ARM retrieval quality for novel classes depends on CLIP embedding similarity, which may degrade for highly domain-specific surgical instruments absent from CLIP's pretraining distribution.
The paper is truncated; full ablation results, inference speed, and failure-case analysis are not available in the provided text.

Relevance to Vision-Language Models¶

This work directly extends CLIP-based VLMs to a challenging medical domain, revealing that standard cross-modal alignment (CLIPSeg/CRIS) is insufficient for fine-grained endoscopic RIS and requires domain-specific adaptations—frequency decomposition and structured attribute retrieval. The attribute retrieval mechanism is a form of structured prompt engineering at inference time, offering a lightweight alternative to fine-tuning for open-vocabulary generalization that is relevant to any VLM deployed in low-annotation, high-specificity domains. The ReferEndoscopy benchmark fills a critical gap for evaluating compositional language grounding in medical VLMs, analogous to RefCOCO/RefCOCO+ for natural images. The frequency-aware fusion strategy—explicitly encoding texture vs. boundary via FFT—is a domain-adaptation technique transferable to other VLM applications where spatial frequency carries semantic meaning.