SOEDiff: Efficient Distillation for Small Object Editing

• URL: http://arxiv.org/abs/2405.09114v1
• Date: Wed, 15 May 2024 06:14:31 GMT
• Title: SOEDiff: Efficient Distillation for Small Object Editing
• Authors: Qihe Pan, Zicheng Wang, Zhen Zhao, Yiming Wu, Sifan Long, Haoran Liang, Ronghua Liang,
• Abstract summary: A new task known as small object editing (SOE) focuses on text-based image inpainting within a constrained, small-sized area. We introduce a novel training-based approach, SOEDiff, aimed at enhancing the capability of baseline models like StableDiffusion in editing small-sized objects. Our method presents significant improvements on the test dataset collected from MSCOCO and OpenImage.
• Score: 9.970035700690744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we delve into a new task known as small object editing (SOE), which focuses on text-based image inpainting within a constrained, small-sized area. Despite the remarkable success have been achieved by current image inpainting approaches, their application to the SOE task generally results in failure cases such as Object Missing, Text-Image Mismatch, and Distortion. These failures stem from the limited use of small-sized objects in training datasets and the downsampling operations employed by U-Net models, which hinders accurate generation. To overcome these challenges, we introduce a novel training-based approach, SOEDiff, aimed at enhancing the capability of baseline models like StableDiffusion in editing small-sized objects while minimizing training costs. Specifically, our method involves two key components: SO-LoRA, which efficiently fine-tunes low-rank matrices, and Cross-Scale Score Distillation loss, which leverages high-resolution predictions from the pre-trained teacher diffusion model. Our method presents significant improvements on the test dataset collected from MSCOCO and OpenImage, validating the effectiveness of our proposed method in small object editing. In particular, when comparing SOEDiff with SD-I model on the OpenImage-f dataset, we observe a 0.99 improvement in CLIP-Score and a reduction of 2.87 in FID. Our project page can be found in https://soediff.github.io/.

Related papers

• Towards Small Object Editing: A Benchmark Dataset and A Training-Free Approach [13.262064234892282]
  Small object generation has been limited due to difficulties in aligning cross-modal attention maps between text and these objects. Our approach offers a training-free method that significantly mitigates this alignment issue with local and global attention guidance. Preliminary results demonstrate the effectiveness of our method, showing marked improvements in the fidelity and accuracy of small object generation compared to existing models.
  arXiv  Detail & Related papers  (2024-11-03T12:38:23Z)
• Looking for Tiny Defects via Forward-Backward Feature Transfer [12.442574943138794]
  We introduce a novel benchmark that evaluates methods on the original, high-resolution image and ground-truth masks. Our benchmark includes a metric that captures robustness with respect to defect size. Our proposal features the highest robustness to defect size, runs at the fastest speed and yields state-of-the-art segmentation performance.
  arXiv  Detail & Related papers  (2024-07-04T17:59:26Z)
• DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
  We propose a training-free method, dubbed DiffUHaul, for the object dragging task. We first apply attention masking in each denoising step to make the generation more disentangled across different objects. In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
  arXiv  Detail & Related papers  (2024-06-03T17:59:53Z)
• Object Detection in Aerial Images in Scarce Data Regimes [0.0]
  Small objects, more numerous in aerial images, are the cause for the apparent performance gap between natural and aerial images. We propose a scale-adaptive box similarity criterion, that improves the training and evaluation of FSOD methods. We also contribute to generic FSOD with two distinct approaches based on metric learning and fine-tuning.
  arXiv  Detail & Related papers  (2023-10-16T14:16:47Z)
• Dense Depth Distillation with Out-of-Distribution Simulated Images [30.79756881887895]
  We study data-free knowledge distillation (KD) for monocular depth estimation (MDE) KD learns a lightweight model for real-world depth perception tasks by compressing it from a trained teacher model while lacking training data in the target domain. We show that our method outperforms the baseline KD by a good margin and even slightly better performance with as few as 1/6 of training images.
  arXiv  Detail & Related papers  (2022-08-26T07:10:01Z)
• Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning [12.697842097171119]
  We present a curriculum learning mechanism that adaptively augments the generated regions, which allows the model to consistently acquire a useful learning signal. Our experiments show that our approach improves on the MoCo v2 baseline by a large margin on multiple object-level tasks when pre-training on multi-object scene image datasets.
  arXiv  Detail & Related papers  (2021-11-26T18:29:57Z)
• To be Critical: Self-Calibrated Weakly Supervised Learning for Salient Object Detection [95.21700830273221]
  Weakly-supervised salient object detection (WSOD) aims to develop saliency models using image-level annotations. We propose a self-calibrated training strategy by explicitly establishing a mutual calibration loop between pseudo labels and network predictions. We prove that even a much smaller dataset with well-matched annotations can facilitate models to achieve better performance as well as generalizability.
  arXiv  Detail & Related papers  (2021-09-04T02:45:22Z)
• Salient Objects in Clutter [130.63976772770368]
  This paper identifies and addresses a serious design bias of existing salient object detection (SOD) datasets. This design bias has led to a saturation in performance for state-of-the-art SOD models when evaluated on existing datasets. We propose a new high-quality dataset and update the previous saliency benchmark.
  arXiv  Detail & Related papers  (2021-05-07T03:49:26Z)
• Few-shot Weakly-Supervised Object Detection via Directional Statistics [55.97230224399744]
  We propose a probabilistic multiple instance learning approach for few-shot Common Object Localization (COL) and few-shot Weakly Supervised Object Detection (WSOD) Our model simultaneously learns the distribution of the novel objects and localizes them via expectation-maximization steps. Our experiments show that the proposed method, despite being simple, outperforms strong baselines in few-shot COL and WSOD, as well as large-scale WSOD tasks.
  arXiv  Detail & Related papers  (2021-03-25T22:34:16Z)
• Mixed-Privacy Forgetting in Deep Networks [114.3840147070712]
  We show that the influence of a subset of the training samples can be removed from the weights of a network trained on large-scale image classification tasks. Inspired by real-world applications of forgetting techniques, we introduce a novel notion of forgetting in mixed-privacy setting. We show that our method allows forgetting without having to trade off the model accuracy.
  arXiv  Detail & Related papers  (2020-12-24T19:34:56Z)
• One-Shot Object Detection without Fine-Tuning [62.39210447209698]
  We introduce a two-stage model consisting of a first stage Matching-FCOS network and a second stage Structure-Aware Relation Module. We also propose novel training strategies that effectively improve detection performance. Our method exceeds the state-of-the-art one-shot performance consistently on multiple datasets.
  arXiv  Detail & Related papers  (2020-05-08T01:59:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.