MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation

• URL: http://arxiv.org/abs/2309.13042v2
• Date: Thu, 03 Oct 2024 20:32:33 GMT
• Title: MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation
• Authors: Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, Chen Change Loy,
• Abstract summary: We present MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation. Our method is training-free and does not rely on any label supervision. Experimental results on the challenging LVIS long-tailed and open-vocabulary benchmarks demonstrate that MosaicFusion can significantly improve the performance of existing instance segmentation models.
• Score: 104.03166324080917
• License:
• Abstract: We present MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation. Our method is training-free and does not rely on any label supervision. Two key designs enable us to employ an off-the-shelf text-to-image diffusion model as a useful dataset generator for object instances and mask annotations. First, we divide an image canvas into several regions and perform a single round of diffusion process to generate multiple instances simultaneously, conditioning on different text prompts. Second, we obtain corresponding instance masks by aggregating cross-attention maps associated with object prompts across layers and diffusion time steps, followed by simple thresholding and edge-aware refinement processing. Without bells and whistles, our MosaicFusion can produce a significant amount of synthetic labeled data for both rare and novel categories. Experimental results on the challenging LVIS long-tailed and open-vocabulary benchmarks demonstrate that MosaicFusion can significantly improve the performance of existing instance segmentation models, especially for rare and novel categories. Code: https://github.com/Jiahao000/MosaicFusion.

Related papers

• DiffuMask-Editor: A Novel Paradigm of Integration Between the Segmentation Diffusion Model and Image Editing to Improve Segmentation Ability [5.767984430681467]
  This paper introduces DiffuMask-Editor, which combines the Diffusion Model for annotated datasets with Image Editing. By integrating multiple objects into images using Text2Image models, our method facilitates the creation of more realistic datasets. Results demonstrate that synthetic data generated by DiffuMask-Editor enable segmentation methods to achieve superior performance compared to real data.
  arXiv  Detail & Related papers  (2024-11-04T05:39:01Z)
• HiDiff: Hybrid Diffusion Framework for Medical Image Segmentation [16.906987804797975]
  HiDiff is a hybrid diffusion framework for medical image segmentation. It can synergize the strengths of existing discriminative segmentation models and new generative diffusion models. It excels at segmenting small objects and generalizing to new datasets.
  arXiv  Detail & Related papers  (2024-07-03T23:59:09Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets. We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
  We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks. To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
  arXiv  Detail & Related papers  (2024-03-25T10:30:22Z)
• Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models [1.6450779686641077]
  We introduce Open-Vocabulary Attention Maps (OVAM)-a training-free method for text-to-image diffusion models. We evaluate these tokens within existing state-of-the-art Stable Diffusion extensions.
  arXiv  Detail & Related papers  (2024-03-21T10:56:12Z)
• DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models [61.906934570771256]
  We present a generic dataset generation model that can produce diverse synthetic images and perception annotations. Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module.
  arXiv  Detail & Related papers  (2023-08-11T14:38:11Z)
• Open-vocabulary Object Segmentation with Diffusion Models [47.36233857830832]
  The goal of this paper is to extract the visual-language correspondence from a pre-trained text-to-image diffusion model, in the form of segmentation map. We adopt the augmented diffusion model to build a synthetic semantic segmentation dataset, and show that, training a standard segmentation model on such dataset demonstrates competitive performance on the zero-shot segmentation(ZS3) benchmark.
  arXiv  Detail & Related papers  (2023-01-12T18:59:08Z)
• SODAR: Segmenting Objects by DynamicallyAggregating Neighboring Mask Representations [90.8752454643737]
  Recent state-of-the-art one-stage instance segmentation model SOLO divides the input image into a grid and directly predicts per grid cell object masks with fully-convolutional networks. We observe SOLO generates similar masks for an object at nearby grid cells, and these neighboring predictions can complement each other as some may better segment certain object part. Motivated by the observed gap, we develop a novel learning-based aggregation method that improves upon SOLO by leveraging the rich neighboring information.
  arXiv  Detail & Related papers  (2022-02-15T13:53:03Z)
• Scaling up instance annotation via label propagation [69.8001043244044]
  We propose a highly efficient annotation scheme for building large datasets with object segmentation masks. We exploit these similarities by using hierarchical clustering on mask predictions made by a segmentation model. We show that we obtain 1M object segmentation masks with a total annotation time of only 290 hours.
  arXiv  Detail & Related papers  (2021-10-05T18:29:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.