Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency

• URL: http://arxiv.org/abs/2307.02516v1
• Date: Wed, 5 Jul 2023 14:28:46 GMT
• Title: Exploring new ways: Enforcing representational dissimilarity to learn new features and reduce error consistency
• Authors: Tassilo Wald and Constantin Ulrich and Fabian Isensee and David Zimmerer and Gregor Koehler and Michael Baumgartner and Klaus H. Maier-Hein
• Abstract summary: We show that highly dissimilar intermediate representations result in less correlated output predictions and slightly lower error consistency. With this, we shine first light on the connection between intermediate representations and their impact on the output predictions.
• Score: 1.7497479054352052
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Independently trained machine learning models tend to learn similar features. Given an ensemble of independently trained models, this results in correlated predictions and common failure modes. Previous attempts focusing on decorrelation of output predictions or logits yielded mixed results, particularly due to their reduction in model accuracy caused by conflicting optimization objectives. In this paper, we propose the novel idea of utilizing methods of the representational similarity field to promote dissimilarity during training instead of measuring similarity of trained models. To this end, we promote intermediate representations to be dissimilar at different depths between architectures, with the goal of learning robust ensembles with disjoint failure modes. We show that highly dissimilar intermediate representations result in less correlated output predictions and slightly lower error consistency, resulting in higher ensemble accuracy. With this, we shine first light on the connection between intermediate representations and their impact on the output predictions.

Related papers

• Dynamic Post-Hoc Neural Ensemblers [55.15643209328513]
  In this study, we explore employing neural networks as ensemble methods. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• DIVE: Subgraph Disagreement for Graph Out-of-Distribution Generalization [44.291382840373]
  This paper addresses the challenge of out-of-distribution generalization in graph machine learning. Traditional graph learning algorithms falter in real-world scenarios where this assumption fails. A principal factor contributing to this suboptimal performance is the inherent simplicity bias of neural networks.
  arXiv  Detail & Related papers  (2024-08-08T12:08:55Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Ensemble Modeling for Multimodal Visual Action Recognition [50.38638300332429]
  We propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset.
  arXiv  Detail & Related papers  (2023-08-10T08:43:20Z)
• Learning Data Representations with Joint Diffusion Models [20.25147743706431]
  Joint machine learning models that allow synthesizing and classifying data often offer uneven performance between those tasks or are unstable to train. We extend the vanilla diffusion model with a classifier that allows for stable joint end-to-end training with shared parameterization between those objectives. The resulting joint diffusion model outperforms recent state-of-the-art hybrid methods in terms of both classification and generation quality on all evaluated benchmarks.
  arXiv  Detail & Related papers  (2023-01-31T13:29:19Z)
• Right for the Right Latent Factors: Debiasing Generative Models via Disentanglement [20.41752850243945]
  Key assumption of most statistical machine learning methods is that they have access to independent samples from the distribution of data they encounter at test time. In particular, machine learning models have been shown to exhibit Clever-Hans-like behaviour, meaning that spurious correlations in the training set are inadvertently learnt. We propose to debias generative models by disentangling their internal representations, which is achieved via human feedback.
  arXiv  Detail & Related papers  (2022-02-01T13:16:18Z)
• Revisiting Consistency Regularization for Semi-Supervised Learning [80.28461584135967]
  We propose an improved consistency regularization framework by a simple yet effective technique, FeatDistLoss. Experimental results show that our model defines a new state of the art for various datasets and settings.
  arXiv  Detail & Related papers  (2021-12-10T20:46:13Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)
• Pattern Similarity-based Machine Learning Methods for Mid-term Load Forecasting: A Comparative Study [0.0]
  We use pattern similarity-based methods for forecasting monthly electricity demand expressing annual seasonality. An integral part of the models is the time series representation using patterns of time series sequences. We consider four such models: nearest neighbor model, fuzzy neighborhood model, kernel regression model and general regression neural network.
  arXiv  Detail & Related papers  (2020-03-03T12:14:36Z)
• An end-to-end approach for the verification problem: learning the right distance [15.553424028461885]
  We augment the metric learning setting by introducing a parametric pseudo-distance, trained jointly with the encoder. We first show it approximates a likelihood ratio which can be used for hypothesis tests. We observe training is much simplified under the proposed approach compared to metric learning with actual distances.
  arXiv  Detail & Related papers  (2020-02-21T18:46:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.