AURA : Automatic Mask Generator using Randomized Input Sampling for Object Removal

• URL: http://arxiv.org/abs/2305.07857v2
• Date: Thu, 03 Oct 2024 18:25:58 GMT
• Title: AURA : Automatic Mask Generator using Randomized Input Sampling for Object Removal
• Authors: Changsuk Oh, H. Jin Kim,
• Abstract summary: In this paper, we focus on generating the input mask to better remove objects using the off-the-shelf image inpainting network. We propose an automatic mask generator inspired by the explainable AI (XAI) method, whose output can better remove objects than a semantic segmentation mask. Experiments confirm that our method shows better performance in removing target class objects than the masks generated from the semantic segmentation maps.
• Score: 26.81218265405809
• License:
• Abstract: The objective of the image inpainting task is to fill missing regions of an image in a visually plausible way. Recently, deep-learning-based image inpainting networks have generated outstanding results, and some utilize their models as object removers by masking unwanted objects in an image. However, while trying to better remove objects using their networks, the previous works pay less attention to the importance of the input mask. In this paper, we focus on generating the input mask to better remove objects using the off-the-shelf image inpainting network. We propose an automatic mask generator inspired by the explainable AI (XAI) method, whose output can better remove objects than a semantic segmentation mask. The proposed method generates an importance map using randomly sampled input masks and quantitatively estimated scores of the completed images obtained from the random masks. The output mask is selected by a judge module among the candidate masks which are generated from the importance map. We design the judge module to quantitatively estimate the quality of the object removal results. In addition, we empirically find that the evaluation methods used in the previous works reporting object removal results are not appropriate for estimating the performance of an object remover. Therefore, we propose new evaluation metrics (FID$^*$ and U-IDS$^*$) to properly evaluate the quality of object removers. Experiments confirm that our method shows better performance in removing target class objects than the masks generated from the semantic segmentation maps, and the two proposed metrics make judgments consistent with humans.

Related papers

• Task-Decoupled Image Inpainting Framework for Class-specific Object Remover [26.81218265405809]
  We propose a task-decoupled image inpainting framework which generates two separate inpainting models. We obtain a class-specific object remover which focuses on removing objects of a target class. Experiments show that the proposed class-specific object remover can better remove target class objects than object removers based on image inpainting networks.
  arXiv  Detail & Related papers  (2024-10-03T18:32:33Z)
• Object Remover Performance Evaluation Methods using Class-wise Object Removal Images [29.28862692741578]
  The performance of an object remover is quantitatively evaluated by measuring the quality of object removal results. We create a dataset with object removal ground truth and compare the evaluations made by the current methods using original images. We propose new evaluation methods tailored to gauge the performance of an object remover.
  arXiv  Detail & Related papers  (2024-04-17T06:40:47Z)
• Inpainting-Driven Mask Optimization for Object Removal [15.429649454099085]
  This paper proposes a mask optimization method for improving the quality of object removal using image inpainting. In our method, this domain gap is resolved by training the inpainting network with object masks extracted by segmentation. To optimize the object masks for inpainting, the segmentation network is connected to the inpainting network and end-to-end trained to improve the inpainting performance.
  arXiv  Detail & Related papers  (2024-03-23T13:52:16Z)
• Variance-insensitive and Target-preserving Mask Refinement for Interactive Image Segmentation [68.16510297109872]
  Point-based interactive image segmentation can ease the burden of mask annotation in applications such as semantic segmentation and image editing. We introduce a novel method, Variance-Insensitive and Target-Preserving Mask Refinement to enhance segmentation quality with fewer user inputs. Experiments on GrabCut, Berkeley, SBD, and DAVIS datasets demonstrate our method's state-of-the-art performance in interactive image segmentation.
  arXiv  Detail & Related papers  (2023-12-22T02:31:31Z)
• Intelligent Debris Mass Estimation Model for Autonomous Underwater Vehicle [0.0]
  Marine debris poses a significant threat to the survival of marine wildlife, often leading to entanglement and starvation. Instance segmentation is an advanced form of object detection that identifies objects and precisely locates and separates them. AUVs use image segmentation to analyze images captured by their cameras to navigate underwater environments.
  arXiv  Detail & Related papers  (2023-09-19T13:47:31Z)
• Inst-Inpaint: Instructing to Remove Objects with Diffusion Models [18.30057229657246]
  In this work, we are interested in an image inpainting algorithm that estimates which object to be removed based on natural language input and removes it, simultaneously. We present a novel inpainting framework, Inst-Inpaint, that can remove objects from images based on the instructions given as text prompts.
  arXiv  Detail & Related papers  (2023-04-06T17:29:50Z)
• Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
  Masked image modeling is a promising self-supervised learning method for visual data. We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:28:55Z)
• Towards Improved Input Masking for Convolutional Neural Networks [66.99060157800403]
  We propose a new masking method for CNNs we call layer masking. We show that our method is able to eliminate or minimize the influence of the mask shape or color on the output of the model. We also demonstrate how the shape of the mask may leak information about the class, thus affecting estimates of model reliance on class-relevant features.
  arXiv  Detail & Related papers  (2022-11-26T19:31:49Z)
• Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation [75.00151934315967]
  MaskDistill is a novel framework for unsupervised semantic segmentation. Our framework does not latch onto low-level image cues and is not limited to object-centric datasets.
  arXiv  Detail & Related papers  (2022-06-13T17:59:43Z)
• What You See is What You Classify: Black Box Attributions [61.998683569022006]
  We train a deep network, the Explainer, to predict attributions for a pre-trained black-box classifier, the Explanandum. Unlike most existing approaches, ours is capable of directly generating very distinct class-specific masks. We show that our attributions are superior to established methods both visually and quantitatively.
  arXiv  Detail & Related papers  (2022-05-23T12:30:04Z)
• High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling [122.06593036862611]
  Existing image inpainting methods often produce artifacts when dealing with large holes in real applications. We propose an iterative inpainting method with a feedback mechanism. Experiments show that our method significantly outperforms existing methods in both quantitative and qualitative evaluations.
  arXiv  Detail & Related papers  (2020-05-24T13:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.