Emergence of Lie symmetries in functional architectures learned by CNNs
- URL: http://arxiv.org/abs/2104.08537v1
- Date: Sat, 17 Apr 2021 13:23:26 GMT
- Title: Emergence of Lie symmetries in functional architectures learned by CNNs
- Authors: Federico Bertoni, Noemi Montobbio, Alessandro Sarti and Giovanna Citti
- Abstract summary: We study the spontaneous development of symmetries in the early layers of a Convolutional Neural Network (CNN) during learning on natural images.
Our architecture is built in such a way to mimic the early stages of biological visual systems.
- Score: 63.69764116066748
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper we study the spontaneous development of symmetries in the early
layers of a Convolutional Neural Network (CNN) during learning on natural
images. Our architecture is built in such a way to mimic the early stages of
biological visual systems. In particular, it contains a pre-filtering step
$\ell^0$ defined in analogy with the Lateral Geniculate Nucleus (LGN).
Moreover, the first convolutional layer is equipped with lateral connections
defined as a propagation driven by a learned connectivity kernel, in analogy
with the horizontal connectivity of the primary visual cortex (V1). The layer
$\ell^0$ shows a rotational symmetric pattern well approximated by a Laplacian
of Gaussian (LoG), which is a well-known model of the receptive profiles of LGN
cells. The convolutional filters in the first layer can be approximated by
Gabor functions, in agreement with well-established models for the profiles of
simple cells in V1. We study the learned lateral connectivity kernel of this
layer, showing the emergence of orientation selectivity w.r.t. the learned
filters. We also examine the association fields induced by the learned kernel,
and show qualitative and quantitative comparisons with known group-based models
of V1 horizontal connectivity. These geometric properties arise spontaneously
during the training of the CNN architecture, analogously to the emergence of
symmetries in visual systems thanks to brain plasticity driven by external
stimuli.
Related papers
- Unveiling the Unseen: Identifiable Clusters in Trained Depthwise
Convolutional Kernels [56.69755544814834]
Recent advances in depthwise-separable convolutional neural networks (DS-CNNs) have led to novel architectures.
This paper reveals another striking property of DS-CNN architectures: discernible and explainable patterns emerge in their trained depthwise convolutional kernels in all layers.
arXiv Detail & Related papers (2024-01-25T19:05:53Z) - Explaining V1 Properties with a Biologically Constrained Deep Learning
Architecture [0.0]
Convolutional neural networks (CNNs) have emerged as promising models of the ventral visual stream, despite their lack of biological specificity.
We show drastic improvements in model-V1 alignment driven by the integration of architectural components that simulate center-surround antagonism, local receptive fields, tuned normalization, and cortical magnification.
Our results highlight an important advancement in the field of NeuroAI, as we systematically establish a set of architectural components that contribute to unprecedented explanation of V1.
arXiv Detail & Related papers (2023-05-18T19:48:05Z) - Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights.
We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
arXiv Detail & Related papers (2023-02-01T03:18:07Z) - Top-down inference in an early visual cortex inspired hierarchical
Variational Autoencoder [0.0]
We exploit advances in Variational Autoencoders to investigate the early visual cortex with sparse coding hierarchical VAEs trained on natural images.
We show that representations similar to the one found in the primary and secondary visual cortices naturally emerge under mild inductive biases.
We show that a neuroscience-inspired choice of the recognition model is critical for two signatures of computations with generative models.
arXiv Detail & Related papers (2022-06-01T12:21:58Z) - Improving Neural Predictivity in the Visual Cortex with Gated Recurrent
Connections [0.0]
We aim to shift the focus on architectures that take into account lateral recurrent connections, a ubiquitous feature of the ventral visual stream, to devise adaptive receptive fields.
In order to increase the robustness of our approach and the biological fidelity of the activations, we employ specific data augmentation techniques.
arXiv Detail & Related papers (2022-03-22T17:27:22Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Neural Cellular Automata Manifold [84.08170531451006]
We show that the neural network architecture of the Neural Cellular Automata can be encapsulated in a larger NN.
This allows us to propose a new model that encodes a manifold of NCA, each of them capable of generating a distinct image.
In biological terms, our approach would play the role of the transcription factors, modulating the mapping of genes into specific proteins that drive cellular differentiation.
arXiv Detail & Related papers (2020-06-22T11:41:57Z) - Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms.
We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
arXiv Detail & Related papers (2020-02-20T15:43:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.