Add-SD: Rational Generation without Manual Reference

• URL: http://arxiv.org/abs/2407.21016v1
• Date: Tue, 30 Jul 2024 17:58:13 GMT
• Title: Add-SD: Rational Generation without Manual Reference
• Authors: Lingfeng Yang, Xinyu Zhang, Xiang Li, Jinwen Chen, Kun Yao, Gang Zhang, Errui Ding, Lingqiao Liu, Jingdong Wang, Jian Yang,
• Abstract summary: We introduce an instruction-based object addition pipeline, named Add-SD, which automatically inserts objects into realistic scenes with rational sizes and positions. Our work contributes in three aspects: proposing a dataset containing numerous instructed image pairs; fine-tuning a diffusion model for rational generation; and generating synthetic data to boost downstream tasks.
• Score: 83.01349699374524
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Diffusion models have exhibited remarkable prowess in visual generalization. Building on this success, we introduce an instruction-based object addition pipeline, named Add-SD, which automatically inserts objects into realistic scenes with rational sizes and positions. Different from layout-conditioned methods, Add-SD is solely conditioned on simple text prompts rather than any other human-costly references like bounding boxes. Our work contributes in three aspects: proposing a dataset containing numerous instructed image pairs; fine-tuning a diffusion model for rational generation; and generating synthetic data to boost downstream tasks. The first aspect involves creating a RemovalDataset consisting of original-edited image pairs with textual instructions, where an object has been removed from the original image while maintaining strong pixel consistency in the background. These data pairs are then used for fine-tuning the Stable Diffusion (SD) model. Subsequently, the pretrained Add-SD model allows for the insertion of expected objects into an image with good rationale. Additionally, we generate synthetic instances for downstream task datasets at scale, particularly for tail classes, to alleviate the long-tailed problem. Downstream tasks benefit from the enriched dataset with enhanced diversity and rationale. Experiments on LVIS val demonstrate that Add-SD yields an improvement of 4.3 mAP on rare classes over the baseline. Code and models are available at https://github.com/ylingfeng/Add-SD.

Related papers

• Decompositional Neural Scene Reconstruction with Generative Diffusion Prior [64.71091831762214]
  Decompositional reconstruction of 3D scenes, with complete shapes and detailed texture, is intriguing for downstream applications. Recent approaches incorporate semantic or geometric regularization to address this issue, but they suffer significant degradation in underconstrained areas. We propose DP-Recon, which employs diffusion priors in the form of Score Distillation Sampling (SDS) to optimize the neural representation of each individual object under novel views.
  arXiv  Detail & Related papers  (2025-03-19T02:11:31Z)
• OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting [54.525583840585305]
  We introduce OmniPaint, a unified framework that re-conceptualizes object removal and insertion as interdependent processes. Our novel CFD metric offers a robust, reference-free evaluation of context consistency and object hallucination.
  arXiv  Detail & Related papers  (2025-03-11T17:55:27Z)
• Adapting Diffusion Models for Improved Prompt Compliance and Controllable Image Synthesis [43.481539150288434]
  This work introduces a new family of. factor graph Diffusion Models (FG-DMs) FG-DMs models the joint distribution of. images and conditioning variables, such as semantic, sketch,. deep or normal maps via a factor graph decomposition.
  arXiv  Detail & Related papers  (2024-10-29T00:54:00Z)
• Unlocking Spatial Comprehension in Text-to-Image Diffusion Models [33.99474729408903]
  CompFuser is an image generation pipeline that enhances spatial comprehension and attribute assignment in text-to-image generative models. Our pipeline enables the interpretation of instructions defining spatial relationships between objects in a scene.
  arXiv  Detail & Related papers  (2023-11-28T19:00:02Z)
• DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models [61.906934570771256]
  We present a generic dataset generation model that can produce diverse synthetic images and perception annotations. Our method builds upon the pre-trained diffusion model and extends text-guided image synthesis to perception data generation. We show that the rich latent code of the diffusion model can be effectively decoded as accurate perception annotations using a decoder module.
  arXiv  Detail & Related papers  (2023-08-11T14:38:11Z)
• Collaborative Score Distillation for Consistent Visual Synthesis [70.29294250371312]
  Collaborative Score Distillation (CSD) is based on the Stein Variational Gradient Descent (SVGD) We show the effectiveness of CSD in a variety of tasks, encompassing the visual editing of panorama images, videos, and 3D scenes. Our results underline the competency of CSD as a versatile method for enhancing inter-sample consistency, thereby broadening the applicability of text-to-image diffusion models.
  arXiv  Detail & Related papers  (2023-07-04T17:31:50Z)
• Expanding Small-Scale Datasets with Guided Imagination [92.5276783917845]
  dataset expansion is a new task aimed at expanding a ready-to-use small dataset by automatically creating new labeled samples. GIF conducts data imagination by optimizing the latent features of the seed data in the semantically meaningful space of the prior model. GIF-SD obtains 13.5% higher model accuracy on natural image datasets than unguided expansion with SD.
  arXiv  Detail & Related papers  (2022-11-25T09:38:22Z)
• Primitive3D: 3D Object Dataset Synthesis from Randomly Assembled Primitives [44.03149443379618]
  We propose a cost-effective method for automatically generating a large amount of 3D objects with annotations. These objects are auto-annotated with part labels originating from primitives. Considering the large overhead of learning on the generated dataset, we propose a dataset distillation strategy.
  arXiv  Detail & Related papers  (2022-05-25T10:07:07Z)
• Lifting 2D Object Locations to 3D by Discounting LiDAR Outliers across Objects and Views [70.1586005070678]
  We present a system for automatically converting 2D mask object predictions and raw LiDAR point clouds into full 3D bounding boxes of objects. Our method significantly outperforms previous work despite the fact that those methods use significantly more complex pipelines, 3D models and additional human-annotated external sources of prior information.
  arXiv  Detail & Related papers  (2021-09-16T13:01:13Z)
• Salient Objects in Clutter [130.63976772770368]
  This paper identifies and addresses a serious design bias of existing salient object detection (SOD) datasets. This design bias has led to a saturation in performance for state-of-the-art SOD models when evaluated on existing datasets. We propose a new high-quality dataset and update the previous saliency benchmark.
  arXiv  Detail & Related papers  (2021-05-07T03:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.