On the benefits of defining vicinal distributions in latent space

• URL: http://arxiv.org/abs/2003.06566v4
• Date: Mon, 18 Oct 2021 09:20:57 GMT
• Title: On the benefits of defining vicinal distributions in latent space
• Abstract summary: Mixup Training (MT) improves the generalization performance of models by introducing globally linear behavior in between training examples. We propose a new approach - textitVarMixup (Variational Mixup) - to better sample mixup images by using the latent manifold underlying the data. Our empirical studies on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that models trained by performing mixup in the latent manifold learned by VAEs are inherently more robust to various input corruptions/perturbations, are significantly better calibrated, and exhibit more local-linear loss landscapes.
• Score: 20.249590931872568
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The vicinal risk minimization (VRM) principle is an empirical risk minimization (ERM) variant that replaces Dirac masses with vicinal functions. There is strong numerical and theoretical evidence showing that VRM outperforms ERM in terms of generalization if appropriate vicinal functions are chosen. Mixup Training (MT), a popular choice of vicinal distribution, improves the generalization performance of models by introducing globally linear behavior in between training examples. Apart from generalization, recent works have shown that mixup trained models are relatively robust to input perturbations/corruptions and at the same time are calibrated better than their non-mixup counterparts. In this work, we investigate the benefits of defining these vicinal distributions like mixup in latent space of generative models rather than in input space itself. We propose a new approach - \textit{VarMixup (Variational Mixup)} - to better sample mixup images by using the latent manifold underlying the data. Our empirical studies on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that models trained by performing mixup in the latent manifold learned by VAEs are inherently more robust to various input corruptions/perturbations, are significantly better calibrated, and exhibit more local-linear loss landscapes.

Related papers

• Hierarchical Exponential-Gaussian Mixtures for Watch-Time Distribution Prediction [74.05279072166697]
  Exponential-Gaussian Mixture Network (EGMN) models the full conditional WT distribution.<n>EGMN is vulnerable to variance collapse, component redundancy, and inactive components.<n>We propose a Hierarchical Exponential-Gaussian Mixture (HEGM) model that addresses these failure modes.
  arXiv  Detail & Related papers  (2026-08-24T15:06:18Z)
• Model Merging via Multi-Teacher Knowledge Distillation [11.543771846135021]
  We introduce a novel flatness-aware PAC-Bayes generalization bound specifically for the model merging setting.<n>We frame model merging as multi-teacher knowledge distillation on scarce, unlabeled data.<n>We formally demonstrate that minimizing the student-teacher Kullback-Leibler divergence directly tightens the upper bound on the merged model's excess risk.
  arXiv  Detail & Related papers  (2025-12-24T17:10:44Z)
• Adaptive Sample-Level Framework Motivated by Distributionally Robust Optimization with Variance-Based Radius Assignment for Enhanced Neural Network Generalization Under Distribution Shift [0.8101875496469488]
  Distribution shifts and minority subpopulations frequently undermine the reliability of deep neural networks trained using Empirical Risk Minimization (ERM)<n>We propose a variance-driven, sample-level DRO framework that automatically identifies high-risk training samples and assigns a personalized robustness budget to each based on its online loss variance.
  arXiv  Detail & Related papers  (2025-11-04T10:20:21Z)
• Towards Understanding Variants of Invariant Risk Minimization through the Lens of Calibration [0.6906005491572401]
  We show that Information Bottleneck-based IRM achieves consistent calibration across different environments. Our empirical evidence indicates that models exhibiting consistent calibration across environments are also well-calibrated.
  arXiv  Detail & Related papers  (2024-01-31T02:08:43Z)
• ClusterDDPM: An EM clustering framework with Denoising Diffusion Probabilistic Models [9.91610928326645]
  Denoising diffusion probabilistic models (DDPMs) represent a new and promising class of generative models. In this study, we introduce an innovative expectation-maximization (EM) framework for clustering using DDPMs. In the M-step, our focus lies in learning clustering-friendly latent representations for the data by employing the conditional DDPM and matching the distribution of latent representations to the mixture of Gaussian priors.
  arXiv  Detail & Related papers  (2023-12-13T10:04:06Z)
• One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls [77.42510898755037]
  One More Step (OMS) is a compact network that incorporates an additional simple yet effective step during inference. OMS elevates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model parameters. Once trained, various pre-trained diffusion models with the same latent domain can share the same OMS module.
  arXiv  Detail & Related papers  (2023-11-27T12:02:42Z)
• Multiclass Alignment of Confidence and Certainty for Network Calibration [10.15706847741555]
  Recent studies reveal that deep neural networks (DNNs) are prone to making overconfident predictions. We propose a new train-time calibration method, which features a simple, plug-and-play auxiliary loss known as multi-class alignment of predictive mean confidence and predictive certainty (MACC) Our method achieves state-of-the-art calibration performance for both in-domain and out-domain predictions.
  arXiv  Detail & Related papers  (2023-09-06T00:56:24Z)
• Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination [33.12164201146458]
  We propose a novel and FL paradigm named FedMR (Federated Model Recombination) The goal of FedMR is to guide the recombined models to be trained towards a flat area. Compared with state-of-the-art FL methods, FedMR can significantly improve the inference accuracy without exposing privacy of each client.
  arXiv  Detail & Related papers  (2023-05-18T05:58:24Z)
• Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
  We propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. The proposed method outperforms existing methods on long-tailed image classification.
  arXiv  Detail & Related papers  (2022-12-02T07:31:39Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalization [89.73665256847858]
  We show that out-of-distribution performance is strongly correlated with in-distribution performance for a wide range of models and distribution shifts. Specifically, we demonstrate strong correlations between in-distribution and out-of-distribution performance on variants of CIFAR-10 & ImageNet. We also investigate cases where the correlation is weaker, for instance some synthetic distribution shifts from CIFAR-10-C and the tissue classification dataset Camelyon17-WILDS.
  arXiv  Detail & Related papers  (2021-07-09T19:48:23Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• Autoregressive Score Matching [113.4502004812927]
  We propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariable log-conditionals (scores) For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
  arXiv  Detail & Related papers  (2020-10-24T07:01:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.