Towards Improved Input Masking for Convolutional Neural Networks

• URL: http://arxiv.org/abs/2211.14646v3
• Date: Sun, 29 Oct 2023 22:11:33 GMT
• Title: Towards Improved Input Masking for Convolutional Neural Networks
• Authors: Sriram Balasubramanian and Soheil Feizi
• Abstract summary: We propose a new masking method for CNNs we call layer masking. We show that our method is able to eliminate or minimize the influence of the mask shape or color on the output of the model. We also demonstrate how the shape of the mask may leak information about the class, thus affecting estimates of model reliance on class-relevant features.
• Score: 66.99060157800403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability to remove features from the input of machine learning models is very important to understand and interpret model predictions. However, this is non-trivial for vision models since masking out parts of the input image typically causes large distribution shifts. This is because the baseline color used for masking (typically grey or black) is out of distribution. Furthermore, the shape of the mask itself can contain unwanted signals which can be used by the model for its predictions. Recently, there has been some progress in mitigating this issue (called missingness bias) in image masking for vision transformers. In this work, we propose a new masking method for CNNs we call layer masking in which the missingness bias caused by masking is reduced to a large extent. Intuitively, layer masking applies a mask to intermediate activation maps so that the model only processes the unmasked input. We show that our method (i) is able to eliminate or minimize the influence of the mask shape or color on the output of the model, and (ii) is much better than replacing the masked region by black or grey for input perturbation based interpretability techniques like LIME. Thus, layer masking is much less affected by missingness bias than other masking strategies. We also demonstrate how the shape of the mask may leak information about the class, thus affecting estimates of model reliance on class-relevant features derived from input masking. Furthermore, we discuss the role of data augmentation techniques for tackling this problem, and argue that they are not sufficient for preventing model reliance on mask shape. The code for this project is publicly available at https://github.com/SriramB-98/layer_masking

Related papers

• Mask Guided Gated Convolution for Amodal Content Completion [0.0]
  We present a model to reconstruct partially visible objects. The model takes a mask as an input, which we call weighted mask. By drawing more attention from the visible region, our model can predict the invisible patch more effectively than the baseline models.
  arXiv  Detail & Related papers  (2024-07-21T15:51:29Z)
• ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders [53.3185750528969]
  Masked AutoEncoders (MAE) have emerged as a robust self-supervised framework. We introduce a data-independent method, termed ColorMAE, which generates different binary mask patterns by filtering random noise. We demonstrate our strategy's superiority in downstream tasks compared to random masking.
  arXiv  Detail & Related papers  (2024-07-17T22:04:00Z)
• SMOOT: Saliency Guided Mask Optimized Online Training [3.024318849346373]
  Saliency-Guided Training (SGT) methods try to highlight the prominent features in the model's training based on the output. SGT makes the model's final result more interpretable by masking input partially. We propose a novel method to determine the optimal number of masked images based on input, accuracy, and model loss during the training.
  arXiv  Detail & Related papers  (2023-10-01T19:41:49Z)
• MP-Former: Mask-Piloted Transformer for Image Segmentation [16.620469868310288]
  Mask2Former suffers from inconsistent mask predictions between decoder layers. We propose a mask-piloted training approach, which feeds noised ground-truth masks in masked-attention and trains the model to reconstruct the original ones.
  arXiv  Detail & Related papers  (2023-03-13T17:57:59Z)
• Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
  Masked image modeling is a promising self-supervised learning method for visual data. We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:28:55Z)
• Masked Autoencoding for Scalable and Generalizable Decision Making [93.84855114717062]
  MaskDP is a simple and scalable self-supervised pretraining method for reinforcement learning and behavioral cloning. We find that a MaskDP model gains the capability of zero-shot transfer to new BC tasks, such as single and multiple goal reaching.
  arXiv  Detail & Related papers  (2022-11-23T07:04:41Z)
• Masked Face Inpainting Through Residual Attention UNet [0.7868449549351486]
  This paper proposes a blind mask face inpainting method using residual attention UNet. A residual block feeds info to the next layer and directly into the layers about two hops away to solve the vanishing gradient problem. Experiments on the publicly available CelebA dataset show the feasibility and robustness of our proposed model.
  arXiv  Detail & Related papers  (2022-09-19T08:49:53Z)
• What You See is What You Classify: Black Box Attributions [61.998683569022006]
  We train a deep network, the Explainer, to predict attributions for a pre-trained black-box classifier, the Explanandum. Unlike most existing approaches, ours is capable of directly generating very distinct class-specific masks. We show that our attributions are superior to established methods both visually and quantitatively.
  arXiv  Detail & Related papers  (2022-05-23T12:30:04Z)
• RePaint: Inpainting using Denoising Diffusion Probabilistic Models [161.74792336127345]
  Free-form inpainting is the task of adding new content to an image in the regions specified by an arbitrary binary mask. We propose RePaint: A Denoising Probabilistic Model (DDPM) based inpainting approach that is applicable to even extreme masks. We validate our method for both faces and general-purpose image inpainting using standard and extreme masks.
  arXiv  Detail & Related papers  (2022-01-24T18:40:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.