From the Loss Landscape to Diverse Feature Learning in Neural Networks

• URL: http://arxiv.org/abs/2608.28948v1
• Date: Fri, 28 Aug 2026 23:38:20 GMT
• Title: From the Loss Landscape to Diverse Feature Learning in Neural Networks
• Abstract summary: Neural networks have grown from an academic curiosity to moving the markets of nations.<n>Despite this explosion in both research and deployment, relatively little is understood about how they achieve the solutions they do.<n>This dissertation elucidates, explains and exploits this special structure in the loss landscape.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Over the course of the last decade, neural networks have grown from an academic curiosity to moving the markets of nations. Despite this explosion in both research and deployment, relatively little is understood about how they achieve the solutions they do. This is both scientifically relevant, and pressing for society. When neural networks make decisions across self-driving, construction, law, hiring and health, there have been and will continue to be unintended consequences. However, attempting to generalize the failures of the largest and most important production systems makes for a very difficult task. Yet signs of these failures exist at all scales of neural networks, so we should be able to study a much more tractable setting. All neural networks must undergo an optimization process, called training, to be useful. To a great degree, understanding neural networks is understanding their optimization: through what process and exposure to which data did they arrive at their results. Yet our knowledge on this topic as a field is quite imprecise. In particular, a curious phenomenon called mode connectivity, the ability to connect neural networks in the loss surface, defies explanation entirely. This dissertation elucidates, explains and exploits this special structure in the loss landscape...

Related papers

• Toward a Physics of Deep Learning and Brains [0.0]
  We show that equations used to describe neuronal avalanches in living brains can also be applied to cascades of activity in deep neural networks.<n>This theoretical framework demonstrates that universal features are shared by both biological and artificial neural networks.
  arXiv  Detail & Related papers  (2025-09-26T17:59:57Z)
• Integrating Causality with Neurochaos Learning: Proposed Approach and Research Agenda [1.534667887016089]
  We investigate how causal and neurochaos learning approaches can be integrated together to produce better results.<n>We propose an approach for this integration to enhance classification, prediction and reinforcement learning.
  arXiv  Detail & Related papers  (2025-01-23T15:45:29Z)
• Simple and Effective Transfer Learning for Neuro-Symbolic Integration [50.592338727912946]
  A potential solution to this issue is Neuro-Symbolic Integration (NeSy), where neural approaches are combined with symbolic reasoning. Most of these methods exploit a neural network to map perceptions to symbols and a logical reasoner to predict the output of the downstream task. They suffer from several issues, including slow convergence, learning difficulties with complex perception tasks, and convergence to local minima. This paper proposes a simple yet effective method to ameliorate these problems.
  arXiv  Detail & Related papers  (2024-02-21T15:51:01Z)
• Benign Overfitting for Two-layer ReLU Convolutional Neural Networks [60.19739010031304]
  We establish algorithm-dependent risk bounds for learning two-layer ReLU convolutional neural networks with label-flipping noise. We show that, under mild conditions, the neural network trained by gradient descent can achieve near-zero training loss and Bayes optimal test risk.
  arXiv  Detail & Related papers  (2023-03-07T18:59:38Z)
• Models Developed for Spiking Neural Networks [0.5801044612920815]
  Spiking neural networks (SNNs) have been around for a long time, and they have been investigated to understand the dynamics of the brain. In this work, we reviewed the structures and performances of SNNs on image classification tasks. The comparisons illustrate that these networks show great capabilities for more complicated problems.
  arXiv  Detail & Related papers  (2022-12-08T16:18:53Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Searching for the Essence of Adversarial Perturbations [73.96215665913797]
  We show that adversarial perturbations contain human-recognizable information, which is the key conspirator responsible for a neural network's erroneous prediction. This concept of human-recognizable information allows us to explain key features related to adversarial perturbations.
  arXiv  Detail & Related papers  (2022-05-30T18:04:57Z)
• Deep Learning and Artificial General Intelligence: Still a Long Way to Go [0.15229257192293197]
  Deep learning using neural network architecture has been on the frontier of computer science research. This article critically shows five major reasons why deep neural networks, as of the current state, are not ready to be the technique of choice for reaching Artificial General Intelligence.
  arXiv  Detail & Related papers  (2022-03-25T23:36:17Z)
• Training Spiking Neural Networks Using Lessons From Deep Learning [28.827506468167652]
  The inner workings of our synapses and neurons provide a glimpse at what the future of deep learning might look like. Some ideas are well accepted and commonly used amongst the neuromorphic engineering community, while others are presented or justified for the first time here. A series of companion interactive tutorials complementary to this paper using our Python package, snnTorch, are also made available.
  arXiv  Detail & Related papers  (2021-09-27T09:28:04Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Adaptive Rational Activations to Boost Deep Reinforcement Learning [68.10769262901003]
  We motivate why rationals are suitable for adaptable activation functions and why their inclusion into neural networks is crucial. We demonstrate that equipping popular algorithms with (recurrent-)rational activations leads to consistent improvements on Atari games.
  arXiv  Detail & Related papers  (2021-02-18T14:53:12Z)
• Graph Structure of Neural Networks [104.33754950606298]
  We show how the graph structure of neural networks affect their predictive performance. A "sweet spot" of relational graphs leads to neural networks with significantly improved predictive performance. Top-performing neural networks have graph structure surprisingly similar to those of real biological neural networks.
  arXiv  Detail & Related papers  (2020-07-13T17:59:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.