Transformations between deep neural networks
- URL: http://arxiv.org/abs/2007.05646v3
- Date: Thu, 14 Jan 2021 16:56:27 GMT
- Title: Transformations between deep neural networks
- Authors: Tom Bertalan and Felix Dietrich and Ioannis G. Kevrekidis
- Abstract summary: We propose to test, and when possible establish, an equivalence between two different artificial neural networks.
We first discuss transformation functions between only the outputs of the two networks.
We then consider transformations that take into account outputs (activations) of a number of internal neurons from each network.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose to test, and when possible establish, an equivalence between two
different artificial neural networks by attempting to construct a data-driven
transformation between them, using manifold-learning techniques. In particular,
we employ diffusion maps with a Mahalanobis-like metric. If the construction
succeeds, the two networks can be thought of as belonging to the same
equivalence class.
We first discuss transformation functions between only the outputs of the two
networks; we then also consider transformations that take into account outputs
(activations) of a number of internal neurons from each network. In general,
Whitney's theorem dictates the number of measurements from one of the networks
required to reconstruct each and every feature of the second network. The
construction of the transformation function relies on a consistent, intrinsic
representation of the network input space.
We illustrate our algorithm by matching neural network pairs trained to learn
(a) observations of scalar functions; (b) observations of two-dimensional
vector fields; and (c) representations of images of a moving three-dimensional
object (a rotating horse). The construction of such equivalence classes across
different network instantiations clearly relates to transfer learning. We also
expect that it will be valuable in establishing equivalence between different
Machine Learning-based models of the same phenomenon observed through different
instruments and by different research groups.
Related papers
- Enhancing lattice kinetic schemes for fluid dynamics with Lattice-Equivariant Neural Networks [79.16635054977068]
We present a new class of equivariant neural networks, dubbed Lattice-Equivariant Neural Networks (LENNs)
Our approach develops within a recently introduced framework aimed at learning neural network-based surrogate models Lattice Boltzmann collision operators.
Our work opens towards practical utilization of machine learning-augmented Lattice Boltzmann CFD in real-world simulations.
arXiv Detail & Related papers (2024-05-22T17:23:15Z) - Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs [35.212818841550835]
One-hidden layer neural networks and one-layer Transformers are studied.
One-hidden layer neural networks attain a maximum $ L_2,k+1 $-margin on a dataset.
We observe similar computational mechanisms in attention of one-layer Transformers.
arXiv Detail & Related papers (2024-02-12T05:52:06Z) - Task structure and nonlinearity jointly determine learned
representational geometry [0.0]
We show that Tanh networks tend to learn representations that reflect the structure of the target outputs, while ReLU networks retain more information about the structure of the raw inputs.
Our findings shed light on the interplay between input-output geometry, nonlinearity, and learned representations in neural networks.
arXiv Detail & Related papers (2024-01-24T16:14:38Z) - Going Beyond Neural Network Feature Similarity: The Network Feature
Complexity and Its Interpretation Using Category Theory [64.06519549649495]
We provide the definition of what we call functionally equivalent features.
These features produce equivalent output under certain transformations.
We propose an efficient algorithm named Iterative Feature Merging.
arXiv Detail & Related papers (2023-10-10T16:27:12Z) - Beyond Geometry: Comparing the Temporal Structure of Computation in
Neural Circuits with Dynamical Similarity Analysis [7.660368798066376]
We introduce a novel similarity metric that compares two systems at the level of their dynamics.
Our method opens the door to comparative analyses of the essential temporal structure of computation in neural circuits.
arXiv Detail & Related papers (2023-06-16T20:11:38Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - A singular Riemannian geometry approach to Deep Neural Networks II.
Reconstruction of 1-D equivalence classes [78.120734120667]
We build the preimage of a point in the output manifold in the input space.
We focus for simplicity on the case of neural networks maps from n-dimensional real spaces to (n - 1)-dimensional real spaces.
arXiv Detail & Related papers (2021-12-17T11:47:45Z) - Similarity and Matching of Neural Network Representations [0.0]
We employ a toolset -- dubbed Dr. Frankenstein -- to analyse the similarity of representations in deep neural networks.
We aim to match the activations on given layers of two trained neural networks by joining them with a stitching layer.
arXiv Detail & Related papers (2021-10-27T17:59:46Z) - Dynamic Inference with Neural Interpreters [72.90231306252007]
We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules.
inputs to the model are routed through a sequence of functions in a way that is end-to-end learned.
We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
arXiv Detail & Related papers (2021-10-12T23:22:45Z) - Learning distinct features helps, provably [98.78384185493624]
We study the diversity of the features learned by a two-layer neural network trained with the least squares loss.
We measure the diversity by the average $L$-distance between the hidden-layer features.
arXiv Detail & Related papers (2021-06-10T19:14:45Z) - Interpretable Neural Networks based classifiers for categorical inputs [0.0]
We introduce a simple way to interpret the output function of a neural network classifier that take as input categorical variables.
We show that in these cases each layer of the network, and the logits layer in particular, can be expanded as a sum of terms that account for the contribution to the classification of each input pattern.
The analysis of the contributions of each pattern, after an appropriate gauge transformation, is presented in two cases where the effectiveness of the method can be appreciated.
arXiv Detail & Related papers (2021-02-05T14:38:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.