Cost-efficient Active Learning for Referring Image Segmentation and Grounding

• URL: http://arxiv.org/abs/2608.30621v2
• Date: Tue, 01 Sep 2026 03:08:18 GMT
• Title: Cost-efficient Active Learning for Referring Image Segmentation and Grounding
• Abstract summary: We tackle the problem of collecting natural-language referring expressions along with region annotations.<n>We introduce Referred Region Ambiguity, a new acquisition function that measures whether the model's confidence collapses onto a single region or disperses across multiple candidates.<n>Our framework consistently outperforms several AL baselines, while a user study shows up to 1.6X faster description labeling of ours.
• Score: 11.362010954029417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Collecting natural-language referring expressions along with region annotations, such as masks or boxes, is a major bottleneck in visual grounding (VG), as annotators must write descriptions that distinguish target regions from visually similar ones. We tackle this by formulating active learning (AL) for VG under the realistic setting where only raw images are available without accompanying text. Since ground-truth text is unavailable, sample selection must estimate which images contain ambiguous regions that would require discriminative referring expressions. To address this, we generate auxiliary region-text pairs using foundation models, and introduce Referred Region Ambiguity, a new acquisition function that measures whether the model's confidence collapses onto a single region or disperses across multiple candidates. It allows our method to prioritize images with strong cross-region competition, which are more informative due to their visual ambiguity. We also design a referring-expression annotation interface that helps annotators quickly focus on writing discriminative language with a few clicks. Experiments on RIS and REC benchmarks show that our AL framework consistently outperforms several AL baselines, while a user study shows up to 1.6X faster description labeling of ours.

Related papers

• Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models [21.17975741743583]
  It has recently been discovered that using a pre-trained vision-language model (VLM), e.g., CLIP, to align a whole query image with several finer text descriptions can significantly enhance zero-shot performance. In this paper, we empirically find that the finer descriptions tend to align more effectively with local areas of the query image rather than the whole image.
  arXiv  Detail & Related papers  (2024-06-05T04:08:41Z)
• Question-Answer Cross Language Image Matching for Weakly Supervised Semantic Segmentation [37.15828464616587]
  Class Activation Map (CAM) has emerged as a popular tool for weakly supervised semantic segmentation. We propose a novel Question-Answer Cross-Language-Image Matching framework for WSSS (QA-CLIMS)
  arXiv  Detail & Related papers  (2024-01-18T10:55:13Z)
• CLIM: Contrastive Language-Image Mosaic for Region Representation [58.05870131126816]
  Contrastive Language-Image Mosaic (CLIM) is a novel approach for aligning region and text representations. CLIM consistently improves different open-vocabulary object detection methods. It can effectively enhance the region representation of vision-language models.
  arXiv  Detail & Related papers  (2023-12-18T17:39:47Z)
• Text Augmented Spatial-aware Zero-shot Referring Image Segmentation [60.84423786769453]
  We introduce a Text Augmented Spatial-aware (TAS) zero-shot referring image segmentation framework. TAS incorporates a mask proposal network for instance-level mask extraction, a text-augmented visual-text matching score for mining the image-text correlation, and a spatial for mask post-processing. The proposed method clearly outperforms state-of-the-art zero-shot referring image segmentation methods.
  arXiv  Detail & Related papers  (2023-10-27T10:52:50Z)
• Referring Image Segmentation Using Text Supervision [44.27304699305985]
  Existing Referring Image (RIS) methods typically require expensive pixel-level or box-level annotations for supervision. We propose a novel weakly-supervised RIS framework to formulate the target localization problem as a classification process. Our framework achieves promising performances to existing fully-supervised RIS methods while outperforming state-of-the-art weakly-supervised methods adapted from related areas.
  arXiv  Detail & Related papers  (2023-08-28T13:40:47Z)
• Advancing Visual Grounding with Scene Knowledge: Benchmark and Method [74.72663425217522]
  Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Most existing VG datasets are constructed using simple description texts. We propose a novel benchmark of underlineScene underlineKnowledge-guided underlineVisual underlineGrounding.
  arXiv  Detail & Related papers  (2023-07-21T13:06:02Z)
• Diffusion Models for Open-Vocabulary Segmentation [79.02153797465324]
  OVDiff is a novel method that leverages generative text-to-image diffusion models for unsupervised open-vocabulary segmentation. It relies solely on pre-trained components and outputs the synthesised segmenter directly, without training.
  arXiv  Detail & Related papers  (2023-06-15T17:51:28Z)
• SpaText: Spatio-Textual Representation for Controllable Image Generation [61.89548017729586]
  SpaText is a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map. We show its effectiveness on two state-of-the-art diffusion models: pixel-based and latent-conditional-based.
  arXiv  Detail & Related papers  (2022-11-25T18:59:10Z)
• RegionCLIP: Region-based Language-Image Pretraining [94.29924084715316]
  Contrastive language-image pretraining (CLIP) using image-text pairs has achieved impressive results on image classification. We propose a new method called RegionCLIP that significantly extends CLIP to learn region-level visual representations. Our method significantly outperforms the state of the art by 3.8 AP50 and 2.2 AP for novel categories on COCO and LVIS datasets.
  arXiv  Detail & Related papers  (2021-12-16T18:39:36Z)
• Detector-Free Weakly Supervised Grounding by Separation [76.65699170882036]
  Weakly Supervised phrase-Grounding (WSG) deals with the task of using data to learn to localize arbitrary text phrases in images. We propose Detector-Free WSG (DF-WSG) to solve WSG without relying on a pre-trained detector. We demonstrate a significant accuracy improvement, of up to $8.5%$ over previous DF-WSG SotA.
  arXiv  Detail & Related papers  (2021-04-20T08:27:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.