Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup

• URL: http://arxiv.org/abs/2109.07177v1
• Date: Wed, 15 Sep 2021 09:31:59 GMT
• Title: Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup
• Abstract summary: Mixup is a recent regularizer for current deep classification networks. We propose the Adversarial Mixing Policy (AMP) to relax the Locally Linear Constraints in Mixup. AMP adds a small adversarial perturbation to the mixing coefficients rather than the examples.
• Score: 7.272501519394588
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mixup is a recent regularizer for current deep classification networks. Through training a neural network on convex combinations of pairs of examples and their labels, it imposes locally linear constraints on the model's input space. However, such strict linear constraints often lead to under-fitting which degrades the effects of regularization. Noticeably, this issue is getting more serious when the resource is extremely limited. To address these issues, we propose the Adversarial Mixing Policy (AMP), organized in a min-max-rand formulation, to relax the Locally Linear Constraints in Mixup. Specifically, AMP adds a small adversarial perturbation to the mixing coefficients rather than the examples. Thus, slight non-linearity is injected in-between the synthetic examples and synthetic labels. By training on these data, the deep networks are further regularized, and thus achieve a lower predictive error rate. Experiments on five text classification benchmarks and five backbone models have empirically shown that our methods reduce the error rate over Mixup variants in a significant margin (up to 31.3%), especially in low-resource conditions (up to 17.5%).

Related papers

• CertMix: Certified, Data-Efficient Metamaterial Design by Affine Mixing of Aligned Neural-Implicit Weight Spaces [6.016090674751934]
  Inverse design of mechanical metamaterials seeks a periodic unit cell whose homogenized elastic properties meet a prescribed target.<n>We introduce CertMix, a data-efficient framework that represents each exemplar unit cell as a small periodic neural implicit field.<n>From as few as 50 exemplars, CertMix attains a scaled property error of $10-4$, roughly two to three orders of magnitude below conditional generative baselines trained on 1000 cells.
  arXiv  Detail & Related papers  (2026-07-05T05:40:01Z)
• Generalization error of min-norm interpolators in transfer learning [2.7309692684728617]
  Min-norm interpolators emerge naturally as implicit regularized limits of modern machine learning algorithms. In many applications, a limited amount of test data may be available during training, yet properties of min-norm in this setting are not well-understood. We establish a novel anisotropic local law to achieve these characterizations.
  arXiv  Detail & Related papers  (2024-06-20T02:23:28Z)
• Fast Semisupervised Unmixing Using Nonconvex Optimization [80.11512905623417]
  We introduce a novel convex convex model for semi/library-based unmixing. We demonstrate the efficacy of Alternating Methods of sparse unsupervised unmixing.
  arXiv  Detail & Related papers  (2024-01-23T10:07:41Z)
• RankMixup: Ranking-Based Mixup Training for Network Calibration [29.994235475637435]
  Network calibration aims to accurately estimate the level of confidences. Recent approaches leverage mixup to calibrate the network's predictions during training. We present RankMixup, a novel mixup-based framework for network calibration.
  arXiv  Detail & Related papers  (2023-08-23T08:25:30Z)
• Over-training with Mixup May Hurt Generalization [32.64382185990981]
  We report a previously unobserved phenomenon in Mixup training. On a number of standard datasets, the performance of Mixup-trained models starts to decay after training for a large number of epochs. We show theoretically that Mixup training may introduce undesired data-dependent label noises to the synthesized data.
  arXiv  Detail & Related papers  (2023-03-02T18:37:34Z)
• Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
  We propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. The proposed method outperforms existing methods on long-tailed image classification.
  arXiv  Detail & Related papers  (2022-12-02T07:31:39Z)
• C-Mixup: Improving Generalization in Regression [71.10418219781575]
  Mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. We propose C-Mixup, which adjusts the sampling probability based on the similarity of the labels. C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively.
  arXiv  Detail & Related papers  (2022-10-11T20:39:38Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• Preventing Manifold Intrusion with Locality: Local Mixup [10.358087436626391]
  Mixup is a data-dependent regularization technique that consists in linearly interpolating input samples and associated outputs. In this paper, we introduce Local Mixup in which distant input samples are weighted down when computing the loss.
  arXiv  Detail & Related papers  (2022-01-12T09:05:53Z)
• Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures [100.55816326422773]
  We study the phenomenon of the maximum margin classifier for linear classification problems. Our results precisely characterize the condition under which benign overfitting can occur.
  arXiv  Detail & Related papers  (2021-04-28T08:25:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.