Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation

• URL: http://arxiv.org/abs/2607.08397v1
• Date: Thu, 09 Jul 2026 12:21:06 GMT
• Title: Attribute Retrieving for Open-Vocabulary Endoscopic Compositional Referring Segmentation
• Abstract summary: Referring Image (RIS) aims to segment image regions specified by natural language, enabling fine-grained and controllable visual understanding.<n>Extending RIS to endoscopic imagery presents unique challenges, including scarce high-quality annotations and complex, domain-specific image-text relationships.<n>We introduce ReferEndoscopy, a large-scale benchmark for RIS in the endoscopy field.<n>We propose the Attribute Retrieval-based Endoscopic-RIS framework for open-vocabulary endoscopic compositional referring segmentation.
• Score: 16.583090298184537
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Referring Image Segmentation (RIS) aims to segment image regions specified by natural language, enabling fine-grained and controllable visual understanding. Extending RIS to endoscopic imagery, however, presents unique challenges, including scarce high-quality annotations and complex, domain-specific image-text relationships. Although recent vision-language models demonstrate strong cross-domain alignment, they often fail to capture fine-grained textual cues in endoscopic settings, resulting in suboptimal performance and limited generalization. To address these challenges, we introduce ReferEndoscopy, a large-scale benchmark for RIS in the endoscopy field. Building on this dataset, we propose the Attribute Retrieval-based Endoscopic-RIS (AR-ERIS) framework for open-vocabulary endoscopic compositional referring segmentation. AR-ERIS leverages attribute retrieval for open-vocabulary endoscopic compositional referring segmentation and is pretrained on the curated ReferEndoscopy dataset, achieving state-of-the-art performance with strong generalization across both simulated and real-world endoscopic data. The dataset and code will be publicly released upon completion of the review process.

Related papers

• EndoVLM: An Endoscopy Vision-Language Pre-training Model via Anatomy-Guided Sparsity and Progressive Alignment [15.371433625713232]
  EndoVLM is a novel vision-language FM pre-trained on over 348K endoscopic examinations.<n>An Anatomy-Guided Sparse Pooling mechanism utilizes textual descriptions as queries to drive sparse attention.<n>A Progressive Semantic-Aware Alignment strategy models clinical taxonomy (anatomy and pathological status) via structured soft targets.<n>A Semantic-Concentrated Masked Autoencoder is applied exclusively to these semantic-rich frames, integrating low-level visual precision with robust high-level semantic representation.
  arXiv  Detail & Related papers  (2026-08-05T05:56:41Z)
• A report-grounded vision-language foundation model for colonoscopy from 280000 routine reports [58.14965296904734]
  EndoCLIP is a vision-language foundation model trained on 125,756 lesion-level image-text pairs.<n>It outperforms general-purpose and biomedical vision-language encoders in both zero-shot and linear-probe settings.
  arXiv  Detail & Related papers  (2026-07-30T16:24:36Z)
• CapCLIP: A Vision-Language Representation Alignment Approach for Wireless Capsule Endoscopy Analysis [0.4551615447454768]
  Wireless capsule endoscopy (WCE) enables non-invasive visual assessment of the small bowel.<n>Existing learning-based approaches for WCE are predominantly vision-only, often confined to narrow pathology sets.<n>This study introduces CapCLIP, a domain-specific vision-language representation learning framework for WCE.
  arXiv  Detail & Related papers  (2026-05-08T21:14:56Z)
• Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline [52.65099689153431]
  Open-vocabulary remote sensing image segmentation (OVRSIS) remains underexplored due to fragmented datasets, limited training diversity, and the lack of evaluation benchmarks.<n>We propose textitOVRSISBenchV2, a large-scale and application-oriented benchmark for OVRSIS.<n>Our results highlight the importance of realistic benchmark design and the effectiveness of perturbation-based transfer for OVRSIS.
  arXiv  Detail & Related papers  (2026-04-17T02:49:46Z)
• Self-Supervised Anatomical Consistency Learning for Vision-Grounded Medical Report Generation [61.350584471060756]
  Vision-grounded medical report generation aims to produce clinically accurate descriptions of medical images.<n>We propose Self-Supervised Anatomical Consistency Learning (SS-ACL) to align generated reports with corresponding anatomical regions.<n>SS-ACL constructs a hierarchical anatomical graph inspired by the invariant top-down inclusion structure of human anatomy.
  arXiv  Detail & Related papers  (2025-09-30T08:59:06Z)
• ACM Multimedia Grand Challenge on ENT Endoscopy Analysis [9.343316855950263]
  We introduce ENTRep, which integrates fine-grained anatomical classification with image-to-image and text-to-image retrieval under bilingual supervision.<n>The dataset comprises expert-annotated images, labeled for anatomical region and normal or abnormal status, and accompanied by dual-language narrative descriptions.
  arXiv  Detail & Related papers  (2025-08-06T18:22:23Z)
• Open Set Recognition for Endoscopic Image Classification: A Deep Learning Approach on the Kvasir Dataset [5.762226441746656]
  We evaluate and compare the OSR capabilities of several representative deep learning architectures, including ResNet-50, Swin Transformer, and a hybrid ResNet-Transformer model, under both closed-set and open-set conditions.<n>This work represents one of the first efforts to apply open set recognition to the Kvasir dataset and provides a benchmark for evaluating OSR performance in medical image analysis.
  arXiv  Detail & Related papers  (2025-06-23T04:39:07Z)
• Referring Expression Instance Retrieval and A Strong End-to-End Baseline [37.47466772169063]
  Text-Image Retrieval retrieves a target image from a gallery based on an image-level description.<n>Referring Expression localizes a target object within a given image using an instance-level description.<n>We introduce a new task called textbfReferring Expression Instance Retrieval (REIR), which supports both instance-level retrieval and localization.
  arXiv  Detail & Related papers  (2025-06-23T02:28:44Z)
• Show and Segment: Universal Medical Image Segmentation via In-Context Learning [43.494896215216684]
  We present Iris, a novel In-context Reference Image guided framework for medical image segmentation.<n>At its core, Iris features a lightweight context task encoding module that distills task-specific information from reference context image-label pairs.<n>By task encoding, Iris supports diverse strategies from one-shot inference and context example ensemble to object-level context example retrieval and in-context.
  arXiv  Detail & Related papers  (2025-03-25T05:26:10Z)
• MRGen: Segmentation Data Engine for Underrepresented MRI Modalities [59.61465292965639]
  Training medical image segmentation models for rare yet clinically important imaging modalities is challenging due to the scarcity of annotated data.<n>This paper investigates leveraging generative models to synthesize data, for training segmentation models for underrepresented modalities.<n>We present MRGen, a data engine for controllable medical image synthesis conditioned on text prompts and segmentation masks.
  arXiv  Detail & Related papers  (2024-12-04T16:34:22Z)
• Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning [65.54680361074882]
  Eye-gaze Guided Multi-modal Alignment (EGMA) framework harnesses eye-gaze data for better alignment of medical visual and textual features. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets.
  arXiv  Detail & Related papers  (2024-03-19T03:59:14Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.