Deep Combinatorial Aggregation
- URL: http://arxiv.org/abs/2210.06436v1
- Date: Wed, 12 Oct 2022 17:35:03 GMT
- Title: Deep Combinatorial Aggregation
- Authors: Yuesong Shen, Daniel Cremers
- Abstract summary: Deep ensemble is a simple and effective method that achieves state-of-the-art results for uncertainty-aware learning tasks.
In this work, we explore a generalization of deep ensemble called deep aggregation (DCA)
DCA creates multiple instances of network components and aggregates their combinations to produce diversified model proposals and predictions.
- Score: 58.78692706974121
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural networks are known to produce poor uncertainty estimations, and a
variety of approaches have been proposed to remedy this issue. This includes
deep ensemble, a simple and effective method that achieves state-of-the-art
results for uncertainty-aware learning tasks. In this work, we explore a
combinatorial generalization of deep ensemble called deep combinatorial
aggregation (DCA). DCA creates multiple instances of network components and
aggregates their combinations to produce diversified model proposals and
predictions. DCA components can be defined at different levels of granularity.
And we discovered that coarse-grain DCAs can outperform deep ensemble for
uncertainty-aware learning both in terms of predictive performance and
uncertainty estimation. For fine-grain DCAs, we discover that an average
parameterization approach named deep combinatorial weight averaging (DCWA) can
improve the baseline training. It is on par with stochastic weight averaging
(SWA) but does not require any custom training schedule or adaptation of
BatchNorm layers. Furthermore, we propose a consistency enforcing loss that
helps the training of DCWA and modelwise DCA. We experiment on in-domain,
distributional shift, and out-of-distribution image classification tasks, and
empirically confirm the effectiveness of DCWA and DCA approaches.
Related papers
- Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning [20.491176017183044]
This paper tackles the multi-objective reinforcement learning (MORL) problem.
It introduces an innovative actor-critic algorithm named MOAC which finds a policy by iteratively making trade-offs among conflicting reward signals.
arXiv Detail & Related papers (2024-05-05T23:52:57Z) - Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep
Learning under Distribution Shift [19.945634052291542]
We evaluate modern BDL algorithms on real-world datasets from the WILDS collection containing challenging classification and regression tasks.
We compare the algorithms on a wide range of large, convolutional and transformer-based neural network architectures.
We provide the first systematic evaluation of BDL for fine-tuning large pre-trained models.
arXiv Detail & Related papers (2023-06-21T14:36:03Z) - Unifying Synergies between Self-supervised Learning and Dynamic
Computation [53.66628188936682]
We present a novel perspective on the interplay between SSL and DC paradigms.
We show that it is feasible to simultaneously learn a dense and gated sub-network from scratch in a SSL setting.
The co-evolution during pre-training of both dense and gated encoder offers a good accuracy-efficiency trade-off.
arXiv Detail & Related papers (2023-01-22T17:12:58Z) - Deep Negative Correlation Classification [82.45045814842595]
Existing deep ensemble methods naively train many different models and then aggregate their predictions.
We propose deep negative correlation classification (DNCC)
DNCC yields a deep classification ensemble where the individual estimator is both accurate and negatively correlated.
arXiv Detail & Related papers (2022-12-14T07:35:20Z) - Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains.
We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
arXiv Detail & Related papers (2021-12-12T06:11:16Z) - Robust Learning via Ensemble Density Propagation in Deep Neural Networks [6.0122901245834015]
We formulate the problem of density propagation through layers of a deep neural network (DNN) and solve it using an Ensemble Density propagation scheme.
Experiments using MNIST and CIFAR-10 datasets show a significant improvement in the robustness of the trained models to random noise and adversarial attacks.
arXiv Detail & Related papers (2021-11-10T21:26:08Z) - Structured DropConnect for Uncertainty Inference in Image Classification [19.22497677849603]
We propose a structured DropConnect (SDC) framework to model the output of a deep neural network by a Dirichlet distribution.
The proposed SDC is adapted well to different network structures with certain generalization capabilities and research prospects.
arXiv Detail & Related papers (2021-06-16T08:31:14Z) - Towards Uncovering the Intrinsic Data Structures for Unsupervised Domain
Adaptation using Structurally Regularized Deep Clustering [119.88565565454378]
Unsupervised domain adaptation (UDA) is to learn classification models that make predictions for unlabeled data on a target domain.
We propose a hybrid model of Structurally Regularized Deep Clustering, which integrates the regularized discriminative clustering of target data with a generative one.
Our proposed H-SRDC outperforms all the existing methods under both the inductive and transductive settings.
arXiv Detail & Related papers (2020-12-08T08:52:00Z) - SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep
Reinforcement Learning [102.78958681141577]
We present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy deep reinforcement learning algorithms.
SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration.
arXiv Detail & Related papers (2020-07-09T17:08:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.