Exploring mechanisms of Neural Robustness: probing the bridge between geometry and spectrum

• URL: http://arxiv.org/abs/2405.00679v1
• Date: Mon, 5 Feb 2024 12:06:00 GMT
• Title: Exploring mechanisms of Neural Robustness: probing the bridge between geometry and spectrum
• Authors: Konstantin Holzhausen, Mia Merlid, Håkon Olav Torvik, Anders Malthe-Sørenssen, Mikkel Elle Lepperød,
• Abstract summary: We study the link between representation smoothness and spectrum by using weight, Jacobian and spectral regularization. Our research aims to understand the interplay between geometry, spectral properties, robustness, and expressivity in neural representations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Backpropagation-optimized artificial neural networks, while precise, lack robustness, leading to unforeseen behaviors that affect their safety. Biological neural systems do solve some of these issues already. Thus, understanding the biological mechanisms of robustness is an important step towards building trustworthy and safe systems. Unlike artificial models, biological neurons adjust connectivity based on neighboring cell activity. Robustness in neural representations is hypothesized to correlate with the smoothness of the encoding manifold. Recent work suggests power law covariance spectra, which were observed studying the primary visual cortex of mice, to be indicative of a balanced trade-off between accuracy and robustness in representations. Here, we show that unsupervised local learning models with winner takes all dynamics learn such power law representations, providing upcoming studies a mechanistic model with that characteristic. Our research aims to understand the interplay between geometry, spectral properties, robustness, and expressivity in neural representations. Hence, we study the link between representation smoothness and spectrum by using weight, Jacobian and spectral regularization while assessing performance and adversarial robustness. Our work serves as a foundation for future research into the mechanisms underlying power law spectra and optimally smooth encodings in both biological and artificial systems. The insights gained may elucidate the mechanisms that realize robust neural networks in mammalian brains and inform the development of more stable and reliable artificial systems.

Related papers

• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment [65.268245109828]
  We examine algorithms for conducting credit assignment in artificial neural networks that are inspired or motivated by neurobiology. We organize the ever-growing set of brain-inspired learning schemes into six general families and consider these in the context of backpropagation of errors. The results of this review are meant to encourage future developments in neuro-mimetic systems and their constituent learning processes.
  arXiv  Detail & Related papers  (2023-12-01T05:20:57Z)
• Learning with Chemical versus Electrical Synapses -- Does it Make a Difference? [61.85704286298537]
  Bio-inspired neural networks have the potential to advance our understanding of neural computation and improve the state-of-the-art of AI systems. We conduct experiments with autonomous lane-keeping through a photorealistic autonomous driving simulator to evaluate their performance under diverse conditions.
  arXiv  Detail & Related papers  (2023-11-21T13:07:20Z)
• What Can the Neural Tangent Kernel Tell Us About Adversarial Robustness? [0.0]
  We study adversarial examples of trained neural networks through analytical tools afforded by recent theory advances connecting neural networks and kernel methods. We show how NTKs allow to generate adversarial examples in a training-free'' fashion, and demonstrate that they transfer to fool their finite-width neural net counterparts in the lazy'' regime.
  arXiv  Detail & Related papers  (2022-10-11T16:11:48Z)
• On the visual analytic intelligence of neural networks [0.463732827131233]
  We present a biologically realistic system that receives inputs from synthetic eye movements - saccades, and processes them with neurons incorporating dynamics of neocortical neurons. We show that the biologically inspired network achieves superior accuracy, learns faster and requires fewer parameters than the conventional network.
  arXiv  Detail & Related papers  (2022-09-28T11:50:29Z)
• Adversarially trained neural representations may already be as robust as corresponding biological neural representations [66.73634912993006]
  We develop a method for performing adversarial visual attacks directly on primate brain activity. We report that the biological neurons that make up visual systems of primates exhibit susceptibility to adversarial perturbations that is comparable in magnitude to existing (robustly trained) artificial neural networks.
  arXiv  Detail & Related papers  (2022-06-19T04:15:29Z)
• Interpretability of Neural Network With Physiological Mechanisms [5.1971653175509145]
  Deep learning continues to play as a powerful state-of-art technique that has achieved extraordinary accuracy levels in various domains of regression and classification tasks. The original goal of proposing the neural network model is to improve the understanding of complex human brains using a mathematical expression approach. Recent deep learning techniques continue to lose the interpretations of its functional process by being treated mostly as a black-box approximator.
  arXiv  Detail & Related papers  (2022-03-24T21:40:04Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception [16.60105791126744]
  We investigate how adversarial perturbations influence the internal representations of visual neural networks. We find distinct geometric signatures for each type of network, revealing different mechanisms for achieving robust representations. Our results shed light on the strategies of robust perception networks, and help explain how geometricality may be beneficial to machine and biological computation.
  arXiv  Detail & Related papers  (2021-11-12T22:59:45Z)
• Optimal input representation in neural systems at the edge of chaos [0.0]
  We build an artificial neural network and train it to classify images. We find that the best performance in such a task is obtained when the network operates near the critical point. We conclude that operating near criticality can have -- besides the usually alleged virtues -- the advantage of allowing for flexible, robust and efficient input representations.
  arXiv  Detail & Related papers  (2021-07-12T19:55:03Z)
• Parsimonious neural networks learn interpretable physical laws [77.34726150561087]
  We propose parsimonious neural networks (PNNs) that combine neural networks with evolutionary optimization to find models that balance accuracy with parsimony. The power and versatility of the approach is demonstrated by developing models for classical mechanics and to predict the melting temperature of materials from fundamental properties.
  arXiv  Detail & Related papers  (2020-05-08T16:15:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.