Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation

• URL: http://arxiv.org/abs/2605.13122v1
• Date: Wed, 13 May 2026 07:48:05 GMT
• Title: Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation
• Abstract summary: We investigate whether language-conditioned visual semantic grounding can be leveraged for zero-shot referring image segmentation.<n>We propose a training-free framework that repurposes pretrained image editing models for RIS by exploiting their intermediate representations.<n>Our approach decomposes localization into two complementary components: attention-based spatial priors that estimate where to focus, and feature-based semantic discrimination that determines what to segment.
• Score: 30.988795633147138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Instruction-based image editing (IIE) models have recently demonstrated strong capability in modifying specific image regions according to natural language instructions, which implicitly requires identifying where an edit should be applied. This indicates that such models inherently perform language-conditioned visual semantic grounding. In this work, we investigate whether this implicit grounding can be leveraged for zero-shot referring image segmentation (RIS), a task that requires pixel-level localization of objects described by natural language expressions. Through systematic analysis, we reveal that strong foreground-background separability emerges in the internal representations of these models at the earliest denoising timestep, well before any visible image transformation occurs. Building on this insight, we propose a training-free framework that repurposes pretrained image editing models for RIS by exploiting their intermediate representations. Our approach decomposes localization into two complementary components: attention-based spatial priors that estimate where to focus, and feature-based semantic discrimination that determines what to segment. By leveraging feature-space separability, the framework produces accurate segmentation masks using only a single denoising step, without requiring full image synthesis. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that our method achieves superior performance over existing zero-shot baselines.

Related papers

• Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation [0.3437656066916039]
  Referring image segmentation aims to produce a pixel-level mask for the image region described by a natural-language expression.<n>We propose SERA, a Spatio-Semantic Expert Routing Architecture for referring image segmentation.<n> SERA introduces lightweight, expression-aware expert refinement at two complementary stages within a vision-language framework.
  arXiv  Detail & Related papers  (2026-03-13T00:37:20Z)
• GS: Generative Segmentation via Label Diffusion [59.380173266566715]
  Language-driven image segmentation is a fundamental task in vision-language understanding, requiring models to segment regions of an image corresponding to natural language expressions.<n>Recent diffusion models have been introduced to this domain, but existing approaches remain image-centric.<n>We propose GS (Generative Label), a novel framework that formulates segmentation itself as a generative task.<n> Experimental results show that GS significantly outperforms existing discriminative and diffusion-based methods, setting a new state-of-the-art for language-driven segmentation.
  arXiv  Detail & Related papers  (2025-08-27T16:28:15Z)
• Deformable Attentive Visual Enhancement for Referring Segmentation Using Vision-Language Model [0.8747606955991707]
  We propose a vision-language model that incorporates architectural improvements to enhance segmentation accuracy and cross-modal alignment.<n>SegVLM shows strong generalization across diverse datasets and referring expression scenarios.
  arXiv  Detail & Related papers  (2025-05-25T17:42:53Z)
• IteRPrimE: Zero-shot Referring Image Segmentation with Iterative Grad-CAM Refinement and Primary Word Emphasis [46.502962768034166]
  Zero-shot Referring Image identifies the instance mask that best aligns with a referring expression without training and fine-tuning.<n>Previous CLIP-based models exhibit a notable reduction in their capacity to discern relative spatial relationships of objects.<n>IteRPrimE outperforms previous state-of-the-art zero-shot methods, particularly excelling in out-of-domain scenarios.
  arXiv  Detail & Related papers  (2025-03-02T15:19:37Z)
• A Simple Framework for Open-Vocabulary Zero-Shot Segmentation [50.58626342189163]
  SimZSS is a framework for open-vocabulary Zero-Shot segmentation.<n>It exploits the discrete nature of text and linguistic knowledge to pinpoint local concepts within captions.<n>SimZSS achieves state-of-the-art results on 7 out of 8 benchmark datasets in less than 15 minutes.
  arXiv  Detail & Related papers  (2024-06-23T11:57:08Z)
• DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
  We propose a training-free method, dubbed DiffUHaul, for the object dragging task. We first apply attention masking in each denoising step to make the generation more disentangled across different objects. In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
  arXiv  Detail & Related papers  (2024-06-03T17:59:53Z)
• Grounding Everything: Emerging Localization Properties in Vision-Language Transformers [51.260510447308306]
  We show that pretrained vision-language (VL) models allow for zero-shot open-vocabulary object localization without any fine-tuning. We propose a Grounding Everything Module (GEM) that generalizes the idea of value-value attention introduced by CLIPSurgery to a self-self attention path. We evaluate the proposed GEM framework on various benchmark tasks and datasets for semantic segmentation.
  arXiv  Detail & Related papers  (2023-12-01T19:06:12Z)
• A Simple Baseline for Zero-shot Semantic Segmentation with Pre-trained Vision-language Model [61.58071099082296]
  It is unclear how to make zero-shot recognition working well on broader vision problems, such as object detection and semantic segmentation. In this paper, we target for zero-shot semantic segmentation, by building it on an off-the-shelf pre-trained vision-language model, i.e., CLIP. Our experimental results show that this simple framework surpasses previous state-of-the-arts by a large margin.
  arXiv  Detail & Related papers  (2021-12-29T18:56:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.