Analysis of Generalizability of Deep Neural Networks Based on the
Complexity of Decision Boundary
- URL: http://arxiv.org/abs/2009.07974v1
- Date: Wed, 16 Sep 2020 23:25:52 GMT
- Title: Analysis of Generalizability of Deep Neural Networks Based on the
Complexity of Decision Boundary
- Authors: Shuyue Guan, Murray Loew
- Abstract summary: We create the decision boundary complexity (DBC) score to define and measure the complexity of decision boundary of deep neural network (DNN) models.
The DBC score is shown to provide an effective method to measure the complexity of a decision boundary and gives a quantitative measure of the generalizability of DNNs.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: For supervised learning models, the analysis of generalization ability
(generalizability) is vital because the generalizability expresses how well a
model will perform on unseen data. Traditional generalization methods, such as
the VC dimension, do not apply to deep neural network (DNN) models. Thus, new
theories to explain the generalizability of DNNs are required. In this study,
we hypothesize that the DNN with a simpler decision boundary has better
generalizability by the law of parsimony (Occam's Razor). We create the
decision boundary complexity (DBC) score to define and measure the complexity
of decision boundary of DNNs. The idea of the DBC score is to generate data
points (called adversarial examples) on or near the decision boundary. Our new
approach then measures the complexity of the boundary using the entropy of
eigenvalues of these data. The method works equally well for high-dimensional
data. We use training data and the trained model to compute the DBC score. And,
the ground truth for model's generalizability is its test accuracy. Experiments
based on the DBC score have verified our hypothesis. The DBC is shown to
provide an effective method to measure the complexity of a decision boundary
and gives a quantitative measure of the generalizability of DNNs.
Related papers
- Generalization of Graph Neural Networks is Robust to Model Mismatch [84.01980526069075]
Graph neural networks (GNNs) have demonstrated their effectiveness in various tasks supported by their generalization capabilities.
In this paper, we examine GNNs that operate on geometric graphs generated from manifold models.
Our analysis reveals the robustness of the GNN generalization in the presence of such model mismatch.
arXiv Detail & Related papers (2024-08-25T16:00:44Z) - On Generalization Bounds for Deep Compound Gaussian Neural Networks [1.4425878137951238]
Unrolled deep neural networks (DNNs) provide better interpretability and superior empirical performance than standard DNNs.
We develop novel generalization error bounds for a class of unrolled DNNs informed by a compound Gaussian prior.
Under realistic conditions, we show that, at worst, the generalization error scales $mathcalO(nsqrt(n))$ in the signal dimension and $mathcalO(($Network Size$)3/2)$ in network size.
arXiv Detail & Related papers (2024-02-20T16:01:39Z) - Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference [5.522612010562183]
Modular-DCM is the first algorithm that, given the causal structure, uses adversarial training to learn the network weights.
We show our algorithm's convergence on the COVIDx dataset and its utility with a causal invariant prediction problem on CelebA-HQ.
arXiv Detail & Related papers (2024-01-02T20:31:15Z) - Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification.
Our results indicate that interpolating with smoother functions leads to better generalization.
arXiv Detail & Related papers (2023-05-30T19:37:44Z) - Validation Diagnostics for SBI algorithms based on Normalizing Flows [55.41644538483948]
This work proposes easy to interpret validation diagnostics for multi-dimensional conditional (posterior) density estimators based on NF.
It also offers theoretical guarantees based on results of local consistency.
This work should help the design of better specified models or drive the development of novel SBI-algorithms.
arXiv Detail & Related papers (2022-11-17T15:48:06Z) - Understanding Deep Learning via Decision Boundary [81.49114762506287]
We show that the neural network with lower decision boundary (DB) variability has better generalizability.
Two new notions, algorithm DB variability and $(epsilon, eta)$-data DB variability, are proposed to measure the decision boundary variability.
arXiv Detail & Related papers (2022-06-03T11:34:12Z) - Confidence Dimension for Deep Learning based on Hoeffding Inequality and
Relative Evaluation [44.393256948610016]
We propose to use multiple factors to measure and rank the relative generalization of deep neural networks (DNNs) based on a new concept of confidence dimension (CD)
Our CD yields a consistent and reliable measure and ranking for both full-precision DNNs and binary neural networks (BNNs) on all the tasks.
arXiv Detail & Related papers (2022-03-17T04:43:43Z) - Post-mortem on a deep learning contest: a Simpson's paradox and the
complementary roles of scale metrics versus shape metrics [61.49826776409194]
We analyze a corpus of models made publicly-available for a contest to predict the generalization accuracy of neural network (NN) models.
We identify what amounts to a Simpson's paradox: where "scale" metrics perform well overall but perform poorly on sub partitions of the data.
We present two novel shape metrics, one data-independent, and the other data-dependent, which can predict trends in the test accuracy of a series of NNs.
arXiv Detail & Related papers (2021-06-01T19:19:49Z) - How to Explain Neural Networks: A perspective of data space division [2.4499092754102874]
Interpretability of algorithms represented by deep learning has been yet an open problem.
We discuss the shortcomings of the existing explainable method based on the two attributes of explanation, which are called completeness and explicitness.
Based on the perspective of the data space division, the principle of complete local interpretable model-agnostic explanations (CLIMEP) is proposed in this paper.
arXiv Detail & Related papers (2021-05-17T13:43:37Z) - Belief Propagation Neural Networks [103.97004780313105]
We introduce belief propagation neural networks (BPNNs)
BPNNs operate on factor graphs and generalize Belief propagation (BP)
We show that BPNNs converges 1.7x faster on Ising models while providing tighter bounds.
On challenging model counting problems, BPNNs compute estimates 100's of times faster than state-of-the-art handcrafted methods.
arXiv Detail & Related papers (2020-07-01T07:39:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.