Diffusion-based image inpainting with internal learning

• URL: http://arxiv.org/abs/2406.04206v1
• Date: Thu, 6 Jun 2024 16:04:06 GMT
• Title: Diffusion-based image inpainting with internal learning
• Authors: Nicolas Cherel, Andrés Almansa, Yann Gousseau, Alasdair Newson,
• Abstract summary: We propose lightweight diffusion models for image inpainting that can be trained on a single image, or a few images. We show that our approach competes with large state-of-the-art models in specific cases.
• Score: 4.912318087940015
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Diffusion models are now the undisputed state-of-the-art for image generation and image restoration. However, they require large amounts of computational power for training and inference. In this paper, we propose lightweight diffusion models for image inpainting that can be trained on a single image, or a few images. We show that our approach competes with large state-of-the-art models in specific cases. We also show that training a model on a single image is particularly relevant for image acquisition modality that differ from the RGB images of standard learning databases. We show results in three different contexts: texture images, line drawing images, and materials BRDF, for which we achieve state-of-the-art results in terms of realism, with a computational load that is greatly reduced compared to concurrent methods.

Related papers

• PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference [62.72779589895124]
  We make the first attempt to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework. We train a reward model with a dataset we construct, consisting of nearly 51,000 images annotated with human preferences. Experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-10-29T11:49:39Z)
• Fast constrained sampling in pre-trained diffusion models [77.21486516041391]
  Diffusion models have dominated the field of large, generative image models. We propose an algorithm for fast-constrained sampling in large pre-trained diffusion models.
  arXiv  Detail & Related papers  (2024-10-24T14:52:38Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets. We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model [80.61157097223058]
  A prevalent strategy to bolster image classification performance is through augmenting the training set with synthetic images generated by T2I models. In this study, we scrutinize the shortcomings of both current generative and conventional data augmentation techniques. We introduce an innovative inter-class data augmentation method known as Diff-Mix, which enriches the dataset by performing image translations between classes.
  arXiv  Detail & Related papers  (2024-03-28T17:23:45Z)
• Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models [53.03978584040557]
  We study image retrieval frameworks that enable us to compare generated images with training samples and detect when content has been replicated. Applying our frameworks to diffusion models trained on multiple datasets including Oxford flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training set size impact rates of content replication.
  arXiv  Detail & Related papers  (2022-12-07T18:58:02Z)
• BlendGAN: Learning and Blending the Internal Distributions of Single Images by Spatial Image-Identity Conditioning [37.21764919074815]
  Single image generative methods are designed to learn the internal patch distribution of a single natural image at multiple scales. We introduce an extended framework, which allows to simultaneously learn the internal distributions of several images. Our BlendGAN opens the door to applications that are not supported by single-image models.
  arXiv  Detail & Related papers  (2022-12-03T10:38:27Z)
• Retrieval-Augmented Diffusion Models [11.278903078792917]
  We propose to complement the diffusion model with a retrieval-based approach and to introduce an explicit memory in the form of an external database. By leveraging CLIP's joint image-text embedding space, our model achieves highly competitive performance on tasks for which it has not been explicitly trained. Our approach incurs low computational and memory overheads and is easy to implement.
  arXiv  Detail & Related papers  (2022-04-25T17:55:26Z)
• Meta Internal Learning [88.68276505511922]
  Internal learning for single-image generation is a framework, where a generator is trained to produce novel images based on a single image. We propose a meta-learning approach that enables training over a collection of images, in order to model the internal statistics of the sample image more effectively. Our results show that the models obtained are as suitable as single-image GANs for many common image applications.
  arXiv  Detail & Related papers  (2021-10-06T16:27:38Z)
• Blind Motion Deblurring through SinGAN Architecture [21.104218472462907]
  Blind motion deblurring involves reconstructing a sharp image from an observation that is blurry. SinGAN is a generative model that is unconditional and could be learned from a single natural image.
  arXiv  Detail & Related papers  (2020-11-07T06:09:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.