Diversity-Aware Agnostic Ensemble of Sharpness Minimizers

• URL: http://arxiv.org/abs/2403.13204v1
• Date: Tue, 19 Mar 2024 23:50:11 GMT
• Title: Diversity-Aware Agnostic Ensemble of Sharpness Minimizers
• Authors: Anh Bui, Vy Vo, Tung Pham, Dinh Phung, Trung Le,
• Abstract summary: We propose DASH - a learning algorithm that promotes diversity and flatness within deep ensembles. We provide a theoretical backbone for our method along with extensive empirical evidence demonstrating an improvement in ensemble generalizability.
• Score: 24.160975100349376
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: There has long been plenty of theoretical and empirical evidence supporting the success of ensemble learning. Deep ensembles in particular take advantage of training randomness and expressivity of individual neural networks to gain prediction diversity, ultimately leading to better generalization, robustness and uncertainty estimation. In respect of generalization, it is found that pursuing wider local minima result in models being more robust to shifts between training and testing sets. A natural research question arises out of these two approaches as to whether a boost in generalization ability can be achieved if ensemble learning and loss sharpness minimization are integrated. Our work investigates this connection and proposes DASH - a learning algorithm that promotes diversity and flatness within deep ensembles. More concretely, DASH encourages base learners to move divergently towards low-loss regions of minimal sharpness. We provide a theoretical backbone for our method along with extensive empirical evidence demonstrating an improvement in ensemble generalizability.

Related papers

• Dynamic Post-Hoc Neural Ensemblers [55.15643209328513]
  In this study, we explore employing neural networks as ensemble methods. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• Multi-Agent Reinforcement Learning from Human Feedback: Data Coverage and Algorithmic Techniques [65.55451717632317]
  We study Multi-Agent Reinforcement Learning from Human Feedback (MARLHF), exploring both theoretical foundations and empirical validations. We define the task as identifying Nash equilibrium from a preference-only offline dataset in general-sum games. Our findings underscore the multifaceted approach required for MARLHF, paving the way for effective preference-based multi-agent systems.
  arXiv  Detail & Related papers  (2024-09-01T13:14:41Z)
• Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance [60.68771286221115]
  We show the interplay between sharpness and diversity within deep ensembles. We introduce SharpBalance, a training approach that balances sharpness and diversity within ensembles. Empirically, we show that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance.
  arXiv  Detail & Related papers  (2024-07-17T20:31:26Z)
• How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalance [64.1656365676171]
  Group imbalance has been a known problem in empirical risk minimization. This paper quantifies the impact of individual groups on the sample complexity, the convergence rate, and the average and group-level testing performance.
  arXiv  Detail & Related papers  (2024-03-12T04:38:05Z)
• Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning [10.784911682565879]
  Ensembling a neural network is a widely recognized approach to enhance model performance, estimate uncertainty, and improve robustness in deep supervised learning. We present a novel self-supervised training regime that leverages an ensemble of independent sub-networks. Our method efficiently builds a sub-model ensemble with high diversity, leading to well-calibrated estimates of model uncertainty.
  arXiv  Detail & Related papers  (2023-08-28T16:58:44Z)
• Diverse Projection Ensembles for Distributional Reinforcement Learning [6.754994171490016]
  This work studies the combination of several different projections and representations in a distributional ensemble. We derive an algorithm that uses ensemble disagreement, measured by the average $1$-Wasserstein distance, as a bonus for deep exploration.
  arXiv  Detail & Related papers  (2023-06-12T13:59:48Z)
• Pathologies of Predictive Diversity in Deep Ensembles [29.893614175153235]
  Classic results establish that encouraging predictive diversity improves performance in ensembles of low-capacity models. Here we demonstrate that these intuitions do not apply to high-capacity neural network ensembles (deep ensembles)
  arXiv  Detail & Related papers  (2023-02-01T19:01:18Z)
• Joint Training of Deep Ensembles Fails Due to Learner Collusion [61.557412796012535]
  Ensembles of machine learning models have been well established as a powerful method of improving performance over a single model. Traditionally, ensembling algorithms train their base learners independently or sequentially with the goal of optimizing their joint performance. We show that directly minimizing the loss of the ensemble appears to rarely be applied in practice.
  arXiv  Detail & Related papers  (2023-01-26T18:58:07Z)
• Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning [79.83792914684985]
  We prove a new identifiability result that provides conditions under which maximally sparse base-predictors yield disentangled representations. Motivated by this theoretical result, we propose a practical approach to learn disentangled representations based on a sparsity-promoting bi-level optimization problem.
  arXiv  Detail & Related papers  (2022-11-26T21:02:09Z)
• Neural Network Ensembles: Theory, Training, and the Importance of Explicit Diversity [6.495473856599276]
  Ensemble learning is a process by which multiple base learners are strategically generated and combined into one composite learner. The right balance of learner accuracy and ensemble diversity can improve the performance of machine learning tasks on benchmark and real-world data sets. Recent theoretical and practical work has demonstrated the subtle trade-off between accuracy and diversity in an ensemble.
  arXiv  Detail & Related papers  (2021-09-29T00:43:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.