GS: Generative Segmentation via Label Diffusion

• URL: http://arxiv.org/abs/2508.20020v1
• Date: Wed, 27 Aug 2025 16:28:15 GMT
• Title: GS: Generative Segmentation via Label Diffusion
• Abstract summary: Language-driven image segmentation is a fundamental task in vision-language understanding, requiring models to segment regions of an image corresponding to natural language expressions.<n>Recent diffusion models have been introduced to this domain, but existing approaches remain image-centric.<n>We propose GS (Generative Label), a novel framework that formulates segmentation itself as a generative task.<n> Experimental results show that GS significantly outperforms existing discriminative and diffusion-based methods, setting a new state-of-the-art for language-driven segmentation.
• Score: 59.380173266566715
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language-driven image segmentation is a fundamental task in vision-language understanding, requiring models to segment regions of an image corresponding to natural language expressions. Traditional methods approach this as a discriminative problem, assigning each pixel to foreground or background based on semantic alignment. Recently, diffusion models have been introduced to this domain, but existing approaches remain image-centric: they either (i) use image diffusion models as visual feature extractors, (ii) synthesize segmentation data via image generation to train discriminative models, or (iii) perform diffusion inversion to extract attention cues from pre-trained image diffusion models-thereby treating segmentation as an auxiliary process. In this paper, we propose GS (Generative Segmentation), a novel framework that formulates segmentation itself as a generative task via label diffusion. Instead of generating images conditioned on label maps and text, GS reverses the generative process: it directly generates segmentation masks from noise, conditioned on both the input image and the accompanying language description. This paradigm makes label generation the primary modeling target, enabling end-to-end training with explicit control over spatial and semantic fidelity. To demonstrate the effectiveness of our approach, we evaluate GS on Panoptic Narrative Grounding (PNG), a representative and challenging benchmark for multimodal segmentation that requires panoptic-level reasoning guided by narrative captions. Experimental results show that GS significantly outperforms existing discriminative and diffusion-based methods, setting a new state-of-the-art for language-driven segmentation.

Related papers

• From Diffusion to Rectified Flow: Rethinking Text-Based Segmentation [6.365160557699766]
  Text-based image segmentation aims to delineate object boundaries within an image from text prompts.<n>Recent studies have shown that diffusion models can provide rich multimodal semantic features.<n>We propose RLFSeg, a novel framework that leverages Rectified Flow to learn direct mapping from the image to the segmentation mask within the latent space.
  arXiv  Detail & Related papers  (2026-05-06T07:40:45Z)
• Diffusion Model as a Generalist Segmentation Learner [44.5756731086797]
  We introduce DiGSeg (Diffusion Models as a Generalist Learner), which repurposes a pretrained diffusion model into a unified framework.<n>A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align queries with evolving visual representations.<n>This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts.
  arXiv  Detail & Related papers  (2026-04-27T15:04:13Z)
• MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generation [89.19760796832765]
  "MagicSeg" is a pipeline for automatically generating datasets tailored to the needs of open-world semantic segmentation.<n>We evaluate our model on PASCAL VOC, PASCAL Context, and COCO, achieving SOTA with performance of 62.9%, 26.7%, and 40.2%, respectively.
  arXiv  Detail & Related papers  (2026-03-20T02:37:38Z)
• Making Training-Free Diffusion Segmentors Scale with the Generative Power [118.72472901404814]
  A line of research focuses on adapting a pre-trained diffusion model to semantic segmentation without any further training.<n>We propose two techniques: auto aggregation and per-pixel rescaling, which together enable training-free segmentation to better leverage generative capability.
  arXiv  Detail & Related papers  (2026-03-06T11:35:37Z)
• Seg-VAR: Image Segmentation with Visual Autoregressive Modeling [60.79579744943664]
  We propose a novel framework that rethinks segmentation as a conditional autoregressive mask generation problem.<n>This is achieved by replacing the discriminative learning with the latent learning process.<n>Our method incorporates three core components: (1) an image encoder generating latent priors from input images, (2) a spatial-aware seglat (a latent expression of segmentation mask) encoder that maps segmentation masks into discrete latent tokens, and (3) a decoder reconstructing masks from these latents.
  arXiv  Detail & Related papers  (2025-11-16T13:36:19Z)
• Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers [56.76198904599581]
  Text-to-image diffusion models excel at translating language prompts into implicitly grounding concepts through their cross-modal attention mechanisms.<n>Recent multi-modal diffusion transformers extend this by introducing joint self-attentiond image and text tokens, enabling richer and more scalable cross-modal alignment.<n>We introduce Seg4Diff, a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image.
  arXiv  Detail & Related papers  (2025-09-22T17:59:54Z)
• Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding [39.73180294057053]
  We propose an Extractive-Injective Phrase Adapter (EIPA) bypass within the Diffusion UNet to dynamically update phrase prompts with image features. We also design a Multi-Level Mutual Aggregation (MLMA) module to reciprocally fuse multi-level image and phrase features for segmentation refinement.
  arXiv  Detail & Related papers  (2024-09-12T17:48:22Z)
• Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation [44.008094698200026]
  FreeDA is a training-free diffusion-augmented method for open-vocabulary semantic segmentation. FreeDA achieves state-of-the-art performance on five datasets.
  arXiv  Detail & Related papers  (2024-04-09T18:00:25Z)
• EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models [52.3015009878545]
  We develop an image segmentor capable of generating fine-grained segmentation maps without any additional training. Our framework identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps. In extensive experiments, the produced segmentation maps are demonstrated to be well delineated and capture detailed parts of the images.
  arXiv  Detail & Related papers  (2024-01-22T07:34:06Z)
• Diffusion Model is Secretly a Training-free Open Vocabulary Semantic Segmenter [47.29967666846132]
  generative text-to-image diffusion models are highly efficient open-vocabulary semantic segmenters. We introduce a novel training-free approach named DiffSegmenter to generate realistic objects that are semantically faithful to the input text. Extensive experiments on three benchmark datasets show that the proposed DiffSegmenter achieves impressive results for open-vocabulary semantic segmentation.
  arXiv  Detail & Related papers  (2023-09-06T06:31:08Z)
• Diffusion Models for Open-Vocabulary Segmentation [79.02153797465324]
  OVDiff is a novel method that leverages generative text-to-image diffusion models for unsupervised open-vocabulary segmentation. It relies solely on pre-trained components and outputs the synthesised segmenter directly, without training.
  arXiv  Detail & Related papers  (2023-06-15T17:51:28Z)
• Stochastic Segmentation with Conditional Categorical Diffusion Models [3.8168879948759953]
  We propose a conditional categorical diffusion model (CCDM) for semantic segmentation based on Denoising Diffusion Probabilistic Models. Our results show that CCDM achieves state-of-the-art performance on LIDC, and outperforms established baselines on the classical segmentation dataset Cityscapes.
  arXiv  Detail & Related papers  (2023-03-15T19:16:47Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks.<n>Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches.<n>We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• Weakly-supervised segmentation of referring expressions [81.73850439141374]
  Text grounded semantic SEGmentation learns segmentation masks directly from image-level referring expressions without pixel-level annotations. Our approach demonstrates promising results for weakly-supervised referring expression segmentation on the PhraseCut and RefCOCO datasets.
  arXiv  Detail & Related papers  (2022-05-10T07:52:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.