Energy-based Dropout in Restricted Boltzmann Machines: Why not go random
- URL: http://arxiv.org/abs/2101.06741v1
- Date: Sun, 17 Jan 2021 18:21:05 GMT
- Title: Energy-based Dropout in Restricted Boltzmann Machines: Why not go random
- Authors: Mateus Roder, Gustavo H. de Rosa, Victor Hugo C. de Albuquerque,
Andr\'e L. D. Rossi, Jo\~ao P. Papa
- Abstract summary: We propose an energy-based Dropout that makes conscious decisions whether a neuron should be dropped or not.
Specifically, we design this regularization method by correlating neurons and the model's energy as an importance level.
The experimental results over several benchmark datasets revealed the proposed approach's suitability compared to the traditional Dropout and the standard RBMs.
- Score: 6.589130992512926
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Deep learning architectures have been widely fostered throughout the last
years, being used in a wide range of applications, such as object recognition,
image reconstruction, and signal processing. Nevertheless, such models suffer
from a common problem known as overfitting, which limits the network from
predicting unseen data effectively. Regularization approaches arise in an
attempt to address such a shortcoming. Among them, one can refer to the
well-known Dropout, which tackles the problem by randomly shutting down a set
of neurons and their connections according to a certain probability. Therefore,
this approach does not consider any additional knowledge to decide which units
should be disconnected. In this paper, we propose an energy-based Dropout
(E-Dropout) that makes conscious decisions whether a neuron should be dropped
or not. Specifically, we design this regularization method by correlating
neurons and the model's energy as an importance level for further applying it
to energy-based models, such as Restricted Boltzmann Machines (RBMs). The
experimental results over several benchmark datasets revealed the proposed
approach's suitability compared to the traditional Dropout and the standard
RBMs.
Related papers
- Efficiency of the hidden fermion determinant states Ansatz in the light of different complexity measures [0.0]
Ans"atze utilizes the expressivity of neural networks to tackle fundamentally challenging problems.
We study five different fermionic models displaying volume law scaling of the entanglement entropy.
We provide evidence that whenever one of the measures indicates proximity to a parameter region in which a conventional approach would work reliable, the neural network approach also works reliable and efficient.
arXiv Detail & Related papers (2024-11-07T08:36:37Z) - On the Benefits of Memory for Modeling Time-Dependent PDEs [35.86010060677811]
We introduce Memory Neural Operator (MemNO), a network based on the recent SSM architectures and Fourier Neural Operator (FNO)
MemNO significantly outperforms the baselines without memory, achieving more than 6 times less error on unseen PDEs.
arXiv Detail & Related papers (2024-09-03T21:56:13Z) - Uncertainty Quantification for Forward and Inverse Problems of PDEs via
Latent Global Evolution [110.99891169486366]
We propose a method that integrates efficient and precise uncertainty quantification into a deep learning-based surrogate model.
Our method endows deep learning-based surrogate models with robust and efficient uncertainty quantification capabilities for both forward and inverse problems.
Our method excels at propagating uncertainty over extended auto-regressive rollouts, making it suitable for scenarios involving long-term predictions.
arXiv Detail & Related papers (2024-02-13T11:22:59Z) - Monotone deep Boltzmann machines [86.50247625239406]
Deep Boltzmann machines (DBMs) are multi-layered probabilistic models governed by a pairwise energy function.
We develop a new class of restricted model, the monotone DBM, which allows for arbitrary self-connection in each layer.
We show that a particular choice of activation results in a fixed-point iteration that gives a variational mean-field solution.
arXiv Detail & Related papers (2023-07-11T03:02:44Z) - Principled Knowledge Extrapolation with GANs [92.62635018136476]
We study counterfactual synthesis from a new perspective of knowledge extrapolation.
We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem.
Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
arXiv Detail & Related papers (2022-05-21T08:39:42Z) - A Survey on Evidential Deep Learning For Single-Pass Uncertainty
Estimation [0.0]
Evidential Deep Learning: For unfamiliar data, they admit "what they don't know" and fall back onto a prior belief.
This survey aims to familiarize the reader with an alternative class of models based on the concept of Evidential Deep Learning: For unfamiliar data, they admit "what they don't know" and fall back onto a prior belief.
arXiv Detail & Related papers (2021-10-06T20:13:57Z) - Discriminator-Free Generative Adversarial Attack [87.71852388383242]
Agenerative-based adversarial attacks can get rid of this limitation.
ASymmetric Saliency-based Auto-Encoder (SSAE) generates the perturbations.
The adversarial examples generated by SSAE not only make thewidely-used models collapse, but also achieves good visual quality.
arXiv Detail & Related papers (2021-07-20T01:55:21Z) - And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks.
We define AND-like neurons and propose measures to increase their proportion in the network.
Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
arXiv Detail & Related papers (2021-02-15T08:19:05Z) - MaxDropout: Deep Neural Network Regularization Based on Maximum Output
Values [0.0]
MaxDropout is a regularizer for deep neural network models that works in a supervised fashion by removing prominent neurons.
We show that it is possible to improve existing neural networks and provide better results in neural networks when Dropout is replaced by MaxDropout.
arXiv Detail & Related papers (2020-07-27T17:55:54Z) - Targeted free energy estimation via learned mappings [66.20146549150475]
Free energy perturbation (FEP) was proposed by Zwanzig more than six decades ago as a method to estimate free energy differences.
FEP suffers from a severe limitation: the requirement of sufficient overlap between distributions.
One strategy to mitigate this problem, called Targeted Free Energy Perturbation, uses a high-dimensional mapping in configuration space to increase overlap.
arXiv Detail & Related papers (2020-02-12T11:10:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.