C-Mixup: Improving Generalization in Regression

• URL: http://arxiv.org/abs/2210.05775v1
• Date: Tue, 11 Oct 2022 20:39:38 GMT
• Title: C-Mixup: Improving Generalization in Regression
• Authors: Huaxiu Yao, Yiping Wang, Linjun Zhang, James Zou, Chelsea Finn
• Abstract summary: Mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. We propose C-Mixup, which adjusts the sampling probability based on the similarity of the labels. C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively.
• Score: 71.10418219781575
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Improving the generalization of deep networks is an important open challenge, particularly in domains without plentiful data. The mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. These interpolated examples augment the original training set. Mixup has shown promising results in various classification tasks, but systematic analysis of mixup in regression remains underexplored. Using mixup directly on regression labels can result in arbitrarily incorrect labels. In this paper, we propose a simple yet powerful algorithm, C-Mixup, to improve generalization on regression tasks. In contrast with vanilla mixup, which picks training examples for mixing with uniform probability, C-Mixup adjusts the sampling probability based on the similarity of the labels. Our theoretical analysis confirms that C-Mixup with label similarity obtains a smaller mean square error in supervised regression and meta-regression than vanilla mixup and using feature similarity. Another benefit of C-Mixup is that it can improve out-of-distribution robustness, where the test distribution is different from the training distribution. By selectively interpolating examples with similar labels, it mitigates the effects of domain-associated information and yields domain-invariant representations. We evaluate C-Mixup on eleven datasets, ranging from tabular to video data. Compared to the best prior approach, C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively. Code is released at https://github.com/huaxiuyao/C-Mixup.

Related papers

• SUMix: Mixup with Semantic and Uncertain Information [41.99721365685618]
  Mixup data augmentation approaches have been applied for various tasks of deep learning. We propose a novel approach named SUMix to learn the mixing ratio as well as the uncertainty for the mixed samples during the training process.
  arXiv  Detail & Related papers  (2024-07-10T16:25:26Z)
• PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis [71.8946280170493]
  This paper introduces PowMix, a versatile embedding space regularizer that builds upon the strengths of unimodal mixing-based regularization approaches. PowMix is integrated before the fusion stage of multimodal architectures and facilitates intra-modal mixing, such as mixing text with text, to act as a regularizer.
  arXiv  Detail & Related papers  (2023-12-19T17:01:58Z)
• Semantic Equivariant Mixup [54.734054770032934]
  Mixup is a well-established data augmentation technique, which can extend the training distribution and regularize the neural networks. Previous mixup variants tend to over-focus on the label-related information. We propose a semantic equivariant mixup (sem) to preserve richer semantic information in the input.
  arXiv  Detail & Related papers  (2023-08-12T03:05:53Z)
• Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup [26.105340203096596]
  We show that non-random selection of pairs affects the training distribution and improve generalization by means completely unrelated to the mixing. We have found a new equivalence between two successful methods: selective mixup and resampling.
  arXiv  Detail & Related papers  (2023-05-26T10:56:22Z)
• Infinite Class Mixup [26.48101652432502]
  Mixup is a strategy for training deep networks where additional samples are augmented by interpolating inputs and labels of training pairs. This paper seeks to address this cornerstone by mixing the classifiers directly instead of mixing the labels for each mixed pair. We show that Infinite Class Mixup outperforms standard Mixup and variants such as RegMixup and Remix on balanced, long-tailed, and data-constrained benchmarks.
  arXiv  Detail & Related papers  (2023-05-17T15:27:35Z)
• Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
  We propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. The proposed method outperforms existing methods on long-tailed image classification.
  arXiv  Detail & Related papers  (2022-12-02T07:31:39Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• MixRL: Data Mixing Augmentation for Regression using Reinforcement Learning [2.1345682889327837]
  Existing techniques for data augmentation largely focus on classification tasks and do not readily apply to regression tasks. We show that mixing examples that either have a large data or label distance may have an increasingly-negative effect on model performance. We propose MixRL, a data augmentation meta learning framework for regression that learns for each example how many nearest neighbors it should be mixed with for the best model performance.
  arXiv  Detail & Related papers  (2021-06-07T07:01:39Z)
• k-Mixup Regularization for Deep Learning via Optimal Transport [32.951696405505686]
  Mixup is a popular regularization technique for training deep neural networks. We extend mixup in a simple, broadly applicable way to emph$k$-mixup, which perturbs $k$-batches of training points in the direction of other $k$-batches. We show that training with $k$-mixup further improves generalization and robustness across several network architectures.
  arXiv  Detail & Related papers  (2021-06-05T17:08:08Z)
• Improving Generalization in Reinforcement Learning with Mixture Regularization [113.12412071717078]
  We introduce a simple approach, named mixreg, which trains agents on a mixture of observations from different training environments. Mixreg increases the data diversity more effectively and helps learn smoother policies. Results show mixreg outperforms the well-established baselines on unseen testing environments by a large margin.
  arXiv  Detail & Related papers  (2020-10-21T08:12:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.