A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective

• URL: http://arxiv.org/abs/2208.09913v1
• Date: Sun, 21 Aug 2022 15:54:25 GMT
• Title: A Unified Analysis of Mixed Sample Data Augmentation: A Loss Function Perspective
• Authors: Chanwoo Park and Sangdoo Yun and Sanghyuk Chun
• Abstract summary: We propose the first unified theoretical analysis of mixed sample data augmentation (MSDA) Our theoretical results show that regardless of the choice of the mixing strategy, MSDA behaves as a pixel-level regularization of the underlying training loss. Our implementation can leverage the advantages of Mixup and CutMix, while our implementation is very efficient.
• Score: 24.33244008451489
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose the first unified theoretical analysis of mixed sample data augmentation (MSDA), such as Mixup and CutMix. Our theoretical results show that regardless of the choice of the mixing strategy, MSDA behaves as a pixel-level regularization of the underlying training loss and a regularization of the first layer parameters. Similarly, our theoretical results support that the MSDA training strategy can improve adversarial robustness and generalization compared to the vanilla training strategy. Using the theoretical results, we provide a high-level understanding of how different design choices of MSDA work differently. For example, we show that the most popular MSDA methods, Mixup and CutMix, behave differently, e.g., CutMix regularizes the input gradients by pixel distances, while Mixup regularizes the input gradients regardless of pixel distances. Our theoretical results also show that the optimal MSDA strategy depends on tasks, datasets, or model parameters. From these observations, we propose generalized MSDAs, a Hybrid version of Mixup and CutMix (HMix) and Gaussian Mixup (GMix), simple extensions of Mixup and CutMix. Our implementation can leverage the advantages of Mixup and CutMix, while our implementation is very efficient, and the computation cost is almost neglectable as Mixup and CutMix. Our empirical study shows that our HMix and GMix outperform the previous state-of-the-art MSDA methods in CIFAR-100 and ImageNet classification tasks. Source code is available at https://github.com/naver-ai/hmix-gmix

Related papers

• SUMix: Mixup with Semantic and Uncertain Information [41.99721365685618]
  Mixup data augmentation approaches have been applied for various tasks of deep learning. We propose a novel approach named SUMix to learn the mixing ratio as well as the uncertainty for the mixed samples during the training process.
  arXiv  Detail & Related papers  (2024-07-10T16:25:26Z)
• PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis [71.8946280170493]
  This paper introduces PowMix, a versatile embedding space regularizer that builds upon the strengths of unimodal mixing-based regularization approaches. PowMix is integrated before the fusion stage of multimodal architectures and facilitates intra-modal mixing, such as mixing text with text, to act as a regularizer.
  arXiv  Detail & Related papers  (2023-12-19T17:01:58Z)
• The Benefits of Mixup for Feature Learning [117.93273337740442]
  We first show that Mixup using different linear parameters for features and labels can still achieve similar performance to standard Mixup. We consider a feature-noise data model and show that Mixup training can effectively learn the rare features from its mixture with the common features. In contrast, standard training can only learn the common features but fails to learn the rare features, thus suffering from bad performance.
  arXiv  Detail & Related papers  (2023-03-15T08:11:47Z)
• MixupE: Understanding and Improving Mixup from Directional Derivative Perspective [86.06981860668424]
  We propose an improved version of Mixup, theoretically justified to deliver better generalization performance than the vanilla Mixup. Our results show that the proposed method improves Mixup across multiple datasets using a variety of architectures.
  arXiv  Detail & Related papers  (2022-12-27T07:03:52Z)
• SMMix: Self-Motivated Image Mixing for Vision Transformers [65.809376136455]
  CutMix is a vital augmentation strategy that determines the performance and generalization ability of vision transformers (ViTs) Existing CutMix variants tackle this problem by generating more consistent mixed images or more precise mixed labels. We propose an efficient and effective Self-Motivated image Mixing method (SMMix) which motivates both image and label enhancement by the model under training itself.
  arXiv  Detail & Related papers  (2022-12-26T00:19:39Z)
• C-Mixup: Improving Generalization in Regression [71.10418219781575]
  Mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. We propose C-Mixup, which adjusts the sampling probability based on the similarity of the labels. C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively.
  arXiv  Detail & Related papers  (2022-10-11T20:39:38Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing [30.14401315979937]
  We propose MixMix to overcome the difficulties of generalizability and inexact inversion. We prove the effectiveness of MixMix from both theoretical and empirical perspectives. MixMix achieves up to 4% and 20% accuracy uplift on quantization and pruning, respectively, compared to existing data-free compression work.
  arXiv  Detail & Related papers  (2020-11-19T15:33:43Z)
• FMix: Enhancing Mixed Sample Data Augmentation [5.820517596386667]
  Mixed Sample Data Augmentation (MSDA) has received increasing attention in recent years. We show that MixUp distorts learned functions in a way that CutMix does not. We propose FMix, an MSDA that uses random binary masks obtained by applying a threshold to low frequency images.
  arXiv  Detail & Related papers  (2020-02-27T11:46:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.