Deep neural networks architectures from the perspective of manifold learning

• URL: http://arxiv.org/abs/2306.03406v1
• Date: Tue, 6 Jun 2023 04:57:39 GMT
• Title: Deep neural networks architectures from the perspective of manifold learning
• Authors: German Magai
• Abstract summary: This paper is a comprehensive comparison and description of neural network architectures in terms of ge-ometry and topology. We focus on the internal representation of neural networks and on the dynamics of changes in the topology and geometry of a data manifold on different layers.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite significant advances in the field of deep learning in ap-plications to various areas, an explanation of the learning pro-cess of neural network models remains an important open ques-tion. The purpose of this paper is a comprehensive comparison and description of neural network architectures in terms of ge-ometry and topology. We focus on the internal representation of neural networks and on the dynamics of changes in the topology and geometry of a data manifold on different layers. In this paper, we use the concepts of topological data analysis (TDA) and persistent homological fractal dimension. We present a wide range of experiments with various datasets and configurations of convolutional neural network (CNNs) architectures and Transformers in CV and NLP tasks. Our work is a contribution to the development of the important field of explainable and interpretable AI within the framework of geometrical deep learning.

Related papers

• Exploring the Manifold of Neural Networks Using Diffusion Geometry [7.038126249994092]
  We learn manifold where datapoints are neural networks by introducing a distance between the hidden layer representations of the neural networks. These distances are then fed to the non-linear dimensionality reduction algorithm PHATE to create a manifold of neural networks. Our analysis reveals that high-performing networks cluster together in the manifold, displaying consistent embedding patterns.
  arXiv  Detail & Related papers  (2024-11-19T16:34:45Z)
• A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes [49.32130498861987]
  We study the case of non-differentiable activation functions, such as ReLU. Two recent works introduced a geometric framework to study neural networks. We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.
  arXiv  Detail & Related papers  (2024-04-09T08:11:46Z)
• Defining Neural Network Architecture through Polytope Structures of Dataset [53.512432492636236]
  This paper defines upper and lower bounds for neural network widths, which are informed by the polytope structure of the dataset in question. We develop an algorithm to investigate a converse situation where the polytope structure of a dataset can be inferred from its corresponding trained neural networks. It is established that popular datasets such as MNIST, Fashion-MNIST, and CIFAR10 can be efficiently encapsulated using no more than two polytopes with a small number of faces.
  arXiv  Detail & Related papers  (2024-02-04T08:57:42Z)
• Probing Biological and Artificial Neural Networks with Task-dependent Neural Manifolds [12.037840490243603]
  We investigate the internal mechanisms of neural networks through the lens of neural population geometry. We quantitatively characterize how different learning objectives lead to differences in the organizational strategies of these models. These analyses present a strong direction for bridging mechanistic and normative theories in neural networks through neural population geometry.
  arXiv  Detail & Related papers  (2023-12-21T20:40:51Z)
• Riemannian Residual Neural Networks [58.925132597945634]
  We show how to extend the residual neural network (ResNet) ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
  arXiv  Detail & Related papers  (2023-10-16T02:12:32Z)
• A Structural Approach to the Design of Domain Specific Neural Network Architectures [0.0]
  This thesis aims to provide a theoretical evaluation of geometric deep learning. It compiles theoretical results that characterize the properties of invariant neural networks with respect to learning performance.
  arXiv  Detail & Related papers  (2023-01-23T11:50:57Z)
• Experimental Observations of the Topology of Convolutional Neural Network Activations [2.4235626091331737]
  Topological data analysis provides compact, noise-robust representations of complex structures. Deep neural networks (DNNs) learn millions of parameters associated with a series of transformations defined by the model architecture. In this paper, we apply cutting edge techniques from TDA with the goal of gaining insight into the interpretability of convolutional neural networks used for image classification.
  arXiv  Detail & Related papers  (2022-12-01T02:05:44Z)
• Topology and geometry of data manifold in deep learning [0.0]
  This article describes and substantiates the geometric and topological view of the learning process of neural networks. We present a wide range of experiments on different datasets and different configurations of convolutional neural network architectures. Our work is a contribution to the development of an important area of explainable and interpretable AI through the example of computer vision.
  arXiv  Detail & Related papers  (2022-04-19T02:57:47Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Inter-layer Information Similarity Assessment of Deep Neural Networks Via Topological Similarity and Persistence Analysis of Data Neighbour Dynamics [93.4221402881609]
  The quantitative analysis of information structure through a deep neural network (DNN) can unveil new insights into the theoretical performance of DNN architectures. Inspired by both LS and ID strategies for quantitative information structure analysis, we introduce two novel complimentary methods for inter-layer information similarity assessment. We demonstrate their efficacy in this study by performing analysis on a deep convolutional neural network architecture on image data.
  arXiv  Detail & Related papers  (2020-12-07T15:34:58Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.