Related papers: A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes

A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes

URL: http://arxiv.org/abs/2404.06104v1
Date: Tue, 9 Apr 2024 08:11:46 GMT
Title: A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes
Authors: Alessandro Benfenati, Alessio Marta,
Abstract summary: We study the case of non-differentiable activation functions, such as ReLU. Two recent works introduced a geometric framework to study neural networks. We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.
Score: 49.32130498861987
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Neural networks are playing a crucial role in everyday life, with the most modern generative models able to achieve impressive results. Nonetheless, their functioning is still not very clear, and several strategies have been adopted to study how and why these model reach their outputs. A common approach is to consider the data in an Euclidean settings: recent years has witnessed instead a shift from this paradigm, moving thus to more general framework, namely Riemannian Geometry. Two recent works introduced a geometric framework to study neural networks making use of singular Riemannian metrics. In this paper we extend these results to convolutional, residual and recursive neural networks, studying also the case of non-differentiable activation functions, such as ReLU. We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.

Related papers

Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
arXiv Detail & Related papers (2024-10-31T22:54:34Z)
A Study on Quantum Graph Neural Networks Applied to Molecular Physics [0.5277756703318045]
This paper introduces a novel architecture for Quantum Graph Neural Networks, which is significantly different from previous approaches found in the literature. The proposed approach produces similar outcomes with respect to previous models but with fewer parameters, resulting in an extremely interpretable architecture rooted in the underlying physics of the problem.
arXiv Detail & Related papers (2024-08-06T20:06:48Z)
Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
arXiv Detail & Related papers (2024-06-14T13:12:07Z)
Riemannian Residual Neural Networks [58.925132597945634]
We show how to extend the residual neural network (ResNet) ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
arXiv Detail & Related papers (2023-10-16T02:12:32Z)
Metric Space Magnitude and Generalisation in Neural Networks [12.110483221042903]
This work is to quantify the learning process of deep neural networks through the lens of a novel topological invariant called magnitude. We use magnitude to study the internal representations of neural networks and propose a new method for determining their generalisation capabilities.
arXiv Detail & Related papers (2023-05-09T17:04:50Z)
Quasi-orthogonality and intrinsic dimensions as measures of learning and generalisation [55.80128181112308]
We show that dimensionality and quasi-orthogonality of neural networks' feature space may jointly serve as network's performance discriminants. Our findings suggest important relationships between the networks' final performance and properties of their randomly initialised feature spaces.
arXiv Detail & Related papers (2022-03-30T21:47:32Z)
A singular Riemannian geometry approach to Deep Neural Networks I. Theoretical foundations [77.86290991564829]
Deep Neural Networks are widely used for solving complex problems in several scientific areas, such as speech recognition, machine translation, image analysis. We study a particular sequence of maps between manifold, with the last manifold of the sequence equipped with a Riemannian metric. We investigate the theoretical properties of the maps of such sequence, eventually we focus on the case of maps between implementing neural networks of practical interest.
arXiv Detail & Related papers (2021-12-17T11:43:30Z)
Hybrid neural network reduced order modelling for turbulent flows with geometric parameters [0.0]
This paper introduces a new technique mixing up a classical Galerkin-projection approach together with a data-driven method to obtain a versatile and accurate algorithm for the resolution of geometrically parametrized incompressible turbulent Navier-Stokes problems. The effectiveness of this procedure is demonstrated on two different test cases: a classical academic back step problem and a shape deformation Ahmed body application.
arXiv Detail & Related papers (2021-07-20T16:06:18Z)
The Loss Surfaces of Neural Networks with General Activation Functions [0.0]
We chart a new path through the spin glass complexity calculations using supersymmetric methods in Random Matrix Theory. Our results shed new light on both the strengths and the weaknesses of spin glass models in this context.
arXiv Detail & Related papers (2020-04-08T12:19:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.