Characterizing the Weight Space for Different Learning Models

• URL: http://arxiv.org/abs/2006.02724v1
• Date: Thu, 4 Jun 2020 09:30:29 GMT
• Title: Characterizing the Weight Space for Different Learning Models
• Authors: Saurav Musunuru, Jay N. Paranjape, Rahul Kumar Dubey and Vijendran G. Venkoparao
• Abstract summary: Deep Learning has become one of the primary research areas in developing intelligent machines. This paper attempts to characterize the solution space of a deep neural network in terms of three different subsets. We show that adversarial attacks are generally less successful against Associative Memory Models than Deep Neural Networks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Learning has become one of the primary research areas in developing intelligent machines. Most of the well-known applications (such as Speech Recognition, Image Processing and NLP) of AI are driven by Deep Learning. Deep Learning algorithms mimic human brain using artificial neural networks and progressively learn to accurately solve a given problem. But there are significant challenges in Deep Learning systems. There have been many attempts to make deep learning models imitate the biological neural network. However, many deep learning models have performed poorly in the presence of adversarial examples. Poor performance in adversarial examples leads to adversarial attacks and in turn leads to safety and security in most of the applications. In this paper we make an attempt to characterize the solution space of a deep neural network in terms of three different subsets viz. weights belonging to exact trained patterns, weights belonging to generalized pattern set and weights belonging to adversarial pattern sets. We attempt to characterize the solution space with two seemingly different learning paradigms viz. the Deep Neural Networks and the Dense Associative Memory Model, which try to achieve learning via quite different mechanisms. We also show that adversarial attacks are generally less successful against Associative Memory Models than Deep Neural Networks.

Related papers

• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• A Survey on State-of-the-art Deep Learning Applications and Challenges [0.0]
  Building a deep learning model is challenging due to the algorithm's complexity and the dynamic nature of real-world problems. This study aims to comprehensively review the state-of-the-art deep learning models in computer vision, natural language processing, time series analysis and pervasive computing.
  arXiv  Detail & Related papers  (2024-03-26T10:10:53Z)
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks [74.3099028063756]
  We develop a new method with neuronal operations based on lateral connections and Hebbian learning. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities. Our method consistently solves for spiking neural networks with nearly zero forgetting.
  arXiv  Detail & Related papers  (2024-02-19T09:29:37Z)
• A Sparse Quantized Hopfield Network for Online-Continual Memory [0.0]
  Nervous systems learn online where a stream of noisy data points are presented in a non-independent, identically distributed (non-i.i.d.) way. Deep networks, on the other hand, typically use non-local learning algorithms and are trained in an offline, non-noisy, i.i.d. setting. We implement this kind of model in a novel neural network called the Sparse Quantized Hopfield Network (SQHN)
  arXiv  Detail & Related papers  (2023-07-27T17:46:17Z)
• Meta-Learning in Spiking Neural Networks with Reward-Modulated STDP [2.179313476241343]
  We propose a bio-plausible meta-learning model inspired by the hippocampus and the prefrontal cortex. Our new model can easily be applied to spike-based neuromorphic devices and enables fast learning in neuromorphic hardware.
  arXiv  Detail & Related papers  (2023-06-07T13:08:46Z)
• Are Deep Neural Networks SMARTer than Second Graders? [85.60342335636341]
  We evaluate the abstraction, deduction, and generalization abilities of neural networks in solving visuo-linguistic puzzles designed for children in the 6--8 age group. Our dataset consists of 101 unique puzzles; each puzzle comprises a picture question, and their solution needs a mix of several elementary skills, including arithmetic, algebra, and spatial reasoning. Experiments reveal that while powerful deep models offer reasonable performances on puzzles in a supervised setting, they are not better than random accuracy when analyzed for generalization.
  arXiv  Detail & Related papers  (2022-12-20T04:33:32Z)
• Continual Learning with Deep Learning Methods in an Application-Oriented Context [0.0]
  An important research area of Artificial Intelligence (AI) deals with the automatic derivation of knowledge from data. One type of machine learning algorithms that can be categorized as "deep learning" model is referred to as Deep Neural Networks (DNNs) DNNs are affected by a problem that prevents new knowledge from being added to an existing base.
  arXiv  Detail & Related papers  (2022-07-12T10:13:33Z)
• Neurosymbolic hybrid approach to driver collision warning [64.02492460600905]
  There are two main algorithmic approaches to autonomous driving systems. Deep learning alone has achieved state-of-the-art results in many areas. But sometimes it can be very difficult to debug if the deep learning model doesn't work.
  arXiv  Detail & Related papers  (2022-03-28T20:29:50Z)
• Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgetting [135.0863818867184]
  artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks. ANV plays as an implicit regularizer of the mutual information between the training data and the learned model. It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
  arXiv  Detail & Related papers  (2020-11-12T06:06:33Z)
• Reservoir Memory Machines as Neural Computers [70.5993855765376]
  Differentiable neural computers extend artificial neural networks with an explicit memory without interference. We achieve some of the computational capabilities of differentiable neural computers with a model that can be trained very efficiently.
  arXiv  Detail & Related papers  (2020-09-14T12:01:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.