FlexiDrop: Theoretical Insights and Practical Advances in Random Dropout Method on GNNs

• URL: http://arxiv.org/abs/2405.20012v1
• Date: Thu, 30 May 2024 12:48:44 GMT
• Title: FlexiDrop: Theoretical Insights and Practical Advances in Random Dropout Method on GNNs
• Authors: Zhiheng Zhou, Sihao Liu, Weichen Zhao,
• Abstract summary: We propose a novel random dropout method for Graph Neural Networks (GNNs) called FlexiDrop. We show that our method enables adaptive adjustment of the dropout rate and theoretically balances the trade-off between model complexity and generalization ability.
• Score: 4.52430575477004
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Graph Neural Networks (GNNs) are powerful tools for handling graph-type data. Recently, GNNs have been widely applied in various domains, but they also face some issues, such as overfitting, over-smoothing and non-robustness. The existing research indicates that random dropout methods are an effective way to address these issues. However, random dropout methods in GNNs still face unresolved problems. Currently, the choice of dropout rate, often determined by heuristic or grid search methods, can increase the generalization error, contradicting the principal aims of dropout. In this paper, we propose a novel random dropout method for GNNs called FlexiDrop. First, we conduct a theoretical analysis of dropout in GNNs using rademacher complexity and demonstrate that the generalization error of traditional random dropout methods is constrained by a function related to the dropout rate. Subsequently, we use this function as a regularizer to unify the dropout rate and empirical loss within a single loss function, optimizing them simultaneously. Therefore, our method enables adaptive adjustment of the dropout rate and theoretically balances the trade-off between model complexity and generalization ability. Furthermore, extensive experimental results on benchmark datasets show that FlexiDrop outperforms traditional random dropout methods in GNNs.

Related papers

• Learning to Reweight for Graph Neural Network [63.978102332612906]
  Graph Neural Networks (GNNs) show promising results for graph tasks. Existing GNNs' generalization ability will degrade when there exist distribution shifts between testing and training graph data. We propose a novel nonlinear graph decorrelation method, which can substantially improve the out-of-distribution generalization ability.
  arXiv  Detail & Related papers  (2023-12-19T12:25:10Z)
• Dropout Reduces Underfitting [85.61466286688385]
  In this study, we demonstrate that dropout can also mitigate underfitting when used at the start of training. We find dropout reduces the directional variance of gradients across mini-batches and helps align the mini-batch gradients with the entire dataset's gradient. Our findings lead us to a solution for improving performance in underfitting models - early dropout: dropout is applied only during the initial phases of training, and turned off afterwards.
  arXiv  Detail & Related papers  (2023-03-02T18:59:15Z)
• Macro-block dropout for improved regularization in training end-to-end speech recognition models [26.06529184341722]
  We define a macro-block that contains a large number of units from the input to a Recurrent Neural Network (RNN) Rather than applying dropout to each unit, we apply random dropout to each macro-block. This algorithm has the effect of applying different drop out rates for each layer even if we keep a constant average dropout rate.
  arXiv  Detail & Related papers  (2022-12-29T02:09:49Z)
• GFlowOut: Dropout with Generative Flow Networks [76.59535235717631]
  Monte Carlo Dropout has been widely used as a relatively cheap way for approximate Inference. Recent works show that the dropout mask can be viewed as a latent variable, which can be inferred with variational inference. GFlowOutleverages the recently proposed probabilistic framework of Generative Flow Networks (GFlowNets) to learn the posterior distribution over dropout masks.
  arXiv  Detail & Related papers  (2022-10-24T03:00:01Z)
• Implicit regularization of dropout [3.42658286826597]
  It is important to understand how dropout, a popular regularization method, aids in achieving a good generalization solution during neural network training. In this work, we present a theoretical derivation of an implicit regularization of dropout, which is validated by a series of experiments. We experimentally find that the training with dropout leads to the neural network with a flatter minimum compared with standard gradient descent training.
  arXiv  Detail & Related papers  (2022-07-13T04:09:14Z)
• DropMessage: Unifying Random Dropping for Graph Neural Networks [9.134120615545866]
  Graph Neural Networks (GNNs) are powerful tools for graph representation learning. Previous works indicate that these problems can be alleviated by random dropping methods, which integrate noises into models by randomly masking parts of the input. We propose a novel random dropping method called DropMessage, which performs dropping operations directly on the message matrix and can be applied to any message-passing GNNs.
  arXiv  Detail & Related papers  (2022-04-21T11:42:49Z)
• Debiased Graph Neural Networks with Agnostic Label Selection Bias [59.61301255860836]
  Most existing Graph Neural Networks (GNNs) are proposed without considering the selection bias in data. We propose a novel Debiased Graph Neural Networks (DGNN) with a differentiated decorrelation regularizer. Our proposed model outperforms the state-of-the-art methods and DGNN is a flexible framework to enhance existing GNNs.
  arXiv  Detail & Related papers  (2022-01-19T16:50:29Z)
• PLACE dropout: A Progressive Layer-wise and Channel-wise Dropout for Domain Generalization [29.824723021053565]
  Domain generalization (DG) aims to learn a generic model from multiple observed source domains. The major challenge in DG is that the model inevitably faces a severe overfitting issue due to the domain gap between source and target domains. We develop a novel layer-wise and channel-wise dropout for DG, which randomly selects one layer and then randomly selects its channels to conduct dropout.
  arXiv  Detail & Related papers  (2021-12-07T13:23:52Z)
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module [60.63525456640462]
  Dropout has been demonstrated as a simple and effective module to regularize the training process of deep neural networks. We propose contextual dropout with an efficient structural design as a simple and scalable sample-dependent dropout module. Our experimental results show that the proposed method outperforms baseline methods in terms of both accuracy and quality of uncertainty estimation.
  arXiv  Detail & Related papers  (2021-03-06T19:30:32Z)
• Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization [62.8384110757689]
  Overfitting ubiquitously exists in real-world applications of deep neural networks (DNNs) The advanced dropout technique applies a model-free and easily implemented distribution with parametric prior, and adaptively adjusts dropout rate. We evaluate the effectiveness of the advanced dropout against nine dropout techniques on seven computer vision datasets.
  arXiv  Detail & Related papers  (2020-10-11T13:19:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.