Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation

• URL: http://arxiv.org/abs/2504.00356v1
• Date: Tue, 01 Apr 2025 02:13:39 GMT
• Title: Hybrid Global-Local Representation with Augmented Spatial Guidance for Zero-Shot Referring Image Segmentation
• Authors: Ting Liu, Siyuan Li,
• Abstract summary: We introduce a training-free, hybrid global-local feature extraction approach that integrates detailed mask-specific features with contextual information from the surrounding area.<n>Experiments on standard RIS benchmarks demonstrate that our method significantly outperforms existing zero-shot RIS models.
• Score: 30.213330400785747
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in zero-shot referring image segmentation (RIS), driven by models such as the Segment Anything Model (SAM) and CLIP, have made substantial progress in aligning visual and textual information. Despite these successes, the extraction of precise and high-quality mask region representations remains a critical challenge, limiting the full potential of RIS tasks. In this paper, we introduce a training-free, hybrid global-local feature extraction approach that integrates detailed mask-specific features with contextual information from the surrounding area, enhancing mask region representation. To further strengthen alignment between mask regions and referring expressions, we propose a spatial guidance augmentation strategy that improves spatial coherence, which is essential for accurately localizing described areas. By incorporating multiple spatial cues, this approach facilitates more robust and precise referring segmentation. Extensive experiments on standard RIS benchmarks demonstrate that our method significantly outperforms existing zero-shot RIS models, achieving substantial performance gains. We believe our approach advances RIS tasks and establishes a versatile framework for region-text alignment, offering broader implications for cross-modal understanding and interaction. Code is available at https://github.com/fhgyuanshen/HybridGL .

Related papers

• HoliSDiP: Image Super-Resolution via Holistic Semantics and Diffusion Prior [62.04939047885834]
  We present HoliSDiP, a framework that leverages semantic segmentation to provide both precise textual and spatial guidance for Real-ISR. Our method employs semantic labels as concise text prompts while introducing dense semantic guidance through segmentation masks and our proposed spatial-CLIP Map.
  arXiv  Detail & Related papers  (2024-11-27T15:22:44Z)
• Region-aware Distribution Contrast: A Novel Approach to Multi-Task Partially Supervised Learning [50.88504784466931]
  Multi-task dense prediction involves semantic segmentation, depth estimation, and surface normal estimation. Existing solutions typically rely on learning global image representations for global cross-task image matching. Our proposal involves modeling region-wise representations using Gaussian Distributions.
  arXiv  Detail & Related papers  (2024-03-15T12:41:30Z)
• RegionGPT: Towards Region Understanding Vision Language Model [88.42271128373191]
  RegionGPT (short as RGPT) is a novel framework designed for complex region-level captioning and understanding. We develop an automated region caption data generation pipeline, enriching the training set with detailed region-level captions. We demonstrate that a universal RGPT model can be effectively applied and significantly enhancing performance across a range of region-level tasks.
  arXiv  Detail & Related papers  (2024-03-04T18:58:08Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• Progressive Feature Self-reinforcement for Weakly Supervised Semantic Segmentation [55.69128107473125]
  We propose a single-stage approach for Weakly Supervised Semantic (WSSS) with image-level labels. We adaptively partition the image content into deterministic regions (e.g., confident foreground and background) and uncertain regions (e.g., object boundaries and misclassified categories) for separate processing. Building upon this, we introduce a complementary self-enhancement method that constrains the semantic consistency between these confident regions and an augmented image with the same class labels.
  arXiv  Detail & Related papers  (2023-12-14T13:21:52Z)
• Towards Generalizable Referring Image Segmentation via Target Prompt and Visual Coherence [48.659338080020746]
  Referring image segmentation (RIS) aims to segment objects in an image conditioning on free-from text descriptions. We present a novel RIS approach, which substantially improves the generalization ability by addressing the two dilemmas mentioned above. Specially, to deal with unconstrained texts, we propose to boost a given expression with an explicit and crucial prompt, which complements the expression in a unified context.
  arXiv  Detail & Related papers  (2023-12-01T09:31:24Z)
• CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentation [29.885991324519463]
  We propose a novel cross-modality masked self-distillation framework named CM-MaskSD. Our method inherits the transferred knowledge of image-text semantic alignment from CLIP model to realize fine-grained patch-word feature alignment. Our framework can considerably boost model performance in a nearly parameter-free manner.
  arXiv  Detail & Related papers  (2023-05-19T07:17:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.