Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision

• URL: http://arxiv.org/abs/2609.06491v1
• Date: Sun, 06 Sep 2026 09:07:02 GMT
• Title: Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision
• Abstract summary: Text-to-image diffusion models can be exploited for open-world entity segmentation without retraining or supervision.<n> Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators.<n>Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000.
• Score: 3.836588693383881
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-world entity segmentation aims to predict masks for arbitrary objects across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. They can therefore be exploited for open-world entity segmentation without retraining or supervision. Building on this observation, we introduce Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in AR_1000 across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on "things" and "stuff+things" datasets and surpasses the detector-based UnSAM on "stuff+things" by 2.1 p.p. in AR_1000. Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, outperforming its fully supervised counterpart with already 5k labeled images.

Related papers

• SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization [64.6372217552272]
  Public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks.<n>We propose SIGMA, which performs semantic-feature differencing in a vision foundation backbone.<n>It produces a 1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets.
  arXiv  Detail & Related papers  (2026-05-27T03:55:13Z)
• Excite, Attend and Segment (EASe): Domain-Agnostic Fine-Grained Mask Discovery with Feature Calibration and Self-Supervised Upsampling [0.0]
  Excite, Attend and Segment (EASe) is an unsupervised domain-agnostic semantic segmentation framework.<n>Our evaluation demonstrates superior performance of EASe over previous state-of-the-arts.
  arXiv  Detail & Related papers  (2026-03-31T22:03:15Z)
• MagicSeg: Open-World Segmentation Pretraining via Counterfactural Diffusion-Based Auto-Generation [89.19760796832765]
  "MagicSeg" is a pipeline for automatically generating datasets tailored to the needs of open-world semantic segmentation.<n>We evaluate our model on PASCAL VOC, PASCAL Context, and COCO, achieving SOTA with performance of 62.9%, 26.7%, and 40.2%, respectively.
  arXiv  Detail & Related papers  (2026-03-20T02:37:38Z)
• Evaluating SAM2 for Video Semantic Segmentation [60.157605818225186]
  The Anything Model 2 (SAM2) has proven to be a powerful foundation model for promptable visual object segmentation in both images and videos.<n>This paper explores the extension of SAM2 to dense Video Semantic (VSS)<n>Our experiments suggest that leveraging SAM2 enhances overall performance in VSS, primarily due to its precise predictions of object boundaries.
  arXiv  Detail & Related papers  (2025-12-01T15:15:16Z)
• LlamaSeg: Image Segmentation via Autoregressive Mask Generation [46.17509085054758]
  We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions.<n>We reformulate image segmentation as a visual generation problem, representing masks as "visual" tokens and employing a LLaMA-style Transformer to predict them directly from image inputs.
  arXiv  Detail & Related papers  (2025-05-26T02:22:41Z)
• E-SAM: Training-Free Segment Every Entity Model [22.29478489117426]
  We introduce E-SAM, a novel training-free framework that exhibits exceptional ES capability.<n>E-SAM achieves state-of-the-art performance compared to prior ES methods, demonstrating a significant improvement by +30.1 on benchmark metrics.
  arXiv  Detail & Related papers  (2025-03-15T11:41:33Z)
• PosSAM: Panoptic Open-vocabulary Segment Anything [58.72494640363136]
  PosSAM is an open-vocabulary panoptic segmentation model that unifies the strengths of the Segment Anything Model (SAM) with the vision-native CLIP model in an end-to-end framework. We introduce a Mask-Aware Selective Ensembling (MASE) algorithm that adaptively enhances the quality of generated masks and boosts the performance of open-vocabulary classification during inference for each image.
  arXiv  Detail & Related papers  (2024-03-14T17:55:03Z)
• Open-Vocabulary Segmentation with Unpaired Mask-Text Supervision [87.15580604023555]
  Unpair-Seg is a novel weakly-supervised open-vocabulary segmentation framework. It learns from unpaired image-mask and image-text pairs, which can be independently and efficiently collected. It achieves 14.6% and 19.5% mIoU on the ADE-847 and PASCAL Context-459 datasets.
  arXiv  Detail & Related papers  (2024-02-14T06:01:44Z)
• Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation [16.863038973001483]
  This work introduces three techniques for diffusion-synthetic semantic segmentation training. First, reliability-aware robust training, originally used in weakly supervised learning, helps segmentation with insufficient synthetic mask quality. Second, large-scale pretraining of whole segmentation models, not only backbones, on synthetic ImageNet-1k-class images with pixel-labels benefits downstream segmentation tasks. Third, we introduce prompt augmentation, data augmentation to the prompt text set to scale up and diversify training images with a limited text resources.
  arXiv  Detail & Related papers  (2023-09-04T05:34:19Z)
• Scaling up instance annotation via label propagation [69.8001043244044]
  We propose a highly efficient annotation scheme for building large datasets with object segmentation masks. We exploit these similarities by using hierarchical clustering on mask predictions made by a segmentation model. We show that we obtain 1M object segmentation masks with a total annotation time of only 290 hours.
  arXiv  Detail & Related papers  (2021-10-05T18:29:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.