k* Distribution: Evaluating the Latent Space of Deep Neural Networks
using Local Neighborhood Analysis
- URL: http://arxiv.org/abs/2312.04024v1
- Date: Thu, 7 Dec 2023 03:42:48 GMT
- Title: k* Distribution: Evaluating the Latent Space of Deep Neural Networks
using Local Neighborhood Analysis
- Authors: Shashank Kotyan, Ueda Tatsuya and Danilo Vasconcellos Vargas
- Abstract summary: dimensionality reduction techniques such as t-SNE or UMAP tend to distort the structure of sample distributions within specific classes in the subset of the latent space.
We introduce the k* Distribution methodology, which focuses on capturing the characteristics and structure of sample distributions for individual classes.
Our study reveals three distinct distributions of samples within the learned latent space subset: a) Fractured, b) Overlapped, and c) Clustered.
- Score: 8.701566919381225
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Most examinations of neural networks' learned latent spaces typically employ
dimensionality reduction techniques such as t-SNE or UMAP. While these methods
effectively capture the overall sample distribution in the entire learned
latent space, they tend to distort the structure of sample distributions within
specific classes in the subset of the latent space. This distortion complicates
the task of easily distinguishing classes identifiable by neural networks. In
response to this challenge, we introduce the k* Distribution methodology. This
approach focuses on capturing the characteristics and structure of sample
distributions for individual classes within the subset of the learned latent
space using local neighborhood analysis. The key concept is to facilitate easy
comparison of different k* distributions, enabling analysis of how various
classes are processed by the same neural network. This provides a more profound
understanding of existing contemporary visualizations. Our study reveals three
distinct distributions of samples within the learned latent space subset: a)
Fractured, b) Overlapped, and c) Clustered. We note and demonstrate that the
distribution of samples within the network's learned latent space significantly
varies depending on the class. Furthermore, we illustrate that our analysis can
be applied to explore the latent space of diverse neural network architectures,
various layers within neural networks, transformations applied to input
samples, and the distribution of training and testing data for neural networks.
We anticipate that our approach will facilitate more targeted investigations
into neural networks by collectively examining the distribution of different
samples within the learned latent space.
Related papers
- An Analytic Solution to Covariance Propagation in Neural Networks [10.013553984400488]
This paper presents a sample-free moment propagation technique to accurately characterize the input-output distributions of neural networks.
A key enabler of our technique is an analytic solution for the covariance of random variables passed through nonlinear activation functions.
The wide applicability and merits of the proposed technique are shown in experiments analyzing the input-output distributions of trained neural networks and training Bayesian neural networks.
arXiv Detail & Related papers (2024-03-24T14:08:24Z) - Topological Representations of Heterogeneous Learning Dynamics of Recurrent Spiking Neural Networks [16.60622265961373]
Spiking Neural Networks (SNNs) have become an essential paradigm in neuroscience and artificial intelligence.
Recent advances in literature have studied the network representations of deep neural networks.
arXiv Detail & Related papers (2024-03-19T05:37:26Z) - Quantile and moment neural networks for learning functionals of
distributions [0.0]
We study news neural networks to approximate function of distributions in a probability space.
Two classes of neural networks based on quantile and moment approximation are proposed to learn these functions.
arXiv Detail & Related papers (2023-03-20T12:23:31Z) - Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights.
We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
arXiv Detail & Related papers (2023-02-01T03:18:07Z) - Discretization Invariant Networks for Learning Maps between Neural
Fields [3.09125960098955]
We present a new framework for understanding and designing discretization invariant neural networks (DI-Nets)
Our analysis establishes upper bounds on the deviation in model outputs under different finite discretizations.
We prove by construction that DI-Nets universally approximate a large class of maps between integrable function spaces.
arXiv Detail & Related papers (2022-06-02T17:44:03Z) - Decomposing neural networks as mappings of correlation functions [57.52754806616669]
We study the mapping between probability distributions implemented by a deep feed-forward network.
We identify essential statistics in the data, as well as different information representations that can be used by neural networks.
arXiv Detail & Related papers (2022-02-10T09:30:31Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Locally Sparse Networks for Interpretable Predictions [7.362415721170984]
We propose a framework for training locally sparse neural networks where the local sparsity is learned via a sample-specific gating mechanism.
The sample-specific sparsity is predicted via a textitgating network, which is trained in tandem with the textitprediction network.
We demonstrate that our method outperforms state-of-the-art models when predicting the target function with far fewer features per instance.
arXiv Detail & Related papers (2021-06-11T15:46:50Z) - A neural anisotropic view of underspecification in deep learning [60.119023683371736]
We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation.
Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
arXiv Detail & Related papers (2021-04-29T14:31:09Z) - Anomaly Detection on Attributed Networks via Contrastive Self-Supervised
Learning [50.24174211654775]
We present a novel contrastive self-supervised learning framework for anomaly detection on attributed networks.
Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair.
A graph neural network-based contrastive learning model is proposed to learn informative embedding from high-dimensional attributes and local structure.
arXiv Detail & Related papers (2021-02-27T03:17:20Z) - Deep Archimedean Copulas [98.96141706464425]
ACNet is a novel differentiable neural network architecture that enforces structural properties.
We show that ACNet is able to both approximate common Archimedean Copulas and generate new copulas which may provide better fits to data.
arXiv Detail & Related papers (2020-12-05T22:58:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.