Stochastic Surprisal: An inferential measurement of Free Energy in Neural Networks

• URL: http://arxiv.org/abs/2302.05776v1
• Date: Sat, 11 Feb 2023 20:41:05 GMT
• Title: Stochastic Surprisal: An inferential measurement of Free Energy in Neural Networks
• Authors: Mohit Prabhushankar and Ghassan AlRegib
• Abstract summary: This paper conjectures and validates a framework that allows for action during inference in supervised neural networks. We introduce a new measurement called surprisal that is a function of the network, the input, and any possible action. We show that, while noise characteristics are ignored to make robust recognition, they are analyzed to estimate quality scores.
• Score: 18.32369721322249
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper conjectures and validates a framework that allows for action during inference in supervised neural networks. Supervised neural networks are constructed with the objective to maximize their performance metric in any given task. This is done by reducing free energy and its associated surprisal during training. However, the bottom-up inference nature of supervised networks is a passive process that renders them fallible to noise. In this paper, we provide a thorough background of supervised neural networks, both generative and discriminative, and discuss their functionality from the perspective of free energy principle. We then provide a framework for introducing action during inference. We introduce a new measurement called stochastic surprisal that is a function of the network, the input, and any possible action. This action can be any one of the outputs that the neural network has learnt, thereby lending stochasticity to the measurement. Stochastic surprisal is validated on two applications: Image Quality Assessment and Recognition under noisy conditions. We show that, while noise characteristics are ignored to make robust recognition, they are analyzed to estimate image quality scores. We apply stochastic surprisal on two applications, three datasets, and as a plug-in on twelve networks. In all, it provides a statistically significant increase among all measures. We conclude by discussing the implications of the proposed stochastic surprisal in other areas of cognitive psychology including expectancy-mismatch and abductive reasoning.

Related papers

• Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes [0.0]
  We propose utilizing the framework of processes, which has been underutilized thus far. We focus solely on activation frequency, leveraging neuroscience techniques used for real neuron spike trains. We derive parameters describing activation patterns in each network, revealing consistent differences across architectures and training sets.
  arXiv  Detail & Related papers  (2023-08-01T22:12:30Z)
• Approximating nonlinear functions with latent boundaries in low-rank excitatory-inhibitory spiking networks [5.955727366271805]
  We put forth a new framework for spike-based excitatory-inhibitory spiking networks. Our work proposes a new perspective on spiking networks that may serve as a starting point for a mechanistic understanding of biological spike-based computation.
  arXiv  Detail & Related papers  (2023-07-18T15:17:00Z)
• Semantic Strengthening of Neuro-Symbolic Learning [85.6195120593625]
  Neuro-symbolic approaches typically resort to fuzzy approximations of a probabilistic objective. We show how to compute this efficiently for tractable circuits. We test our approach on three tasks: predicting a minimum-cost path in Warcraft, predicting a minimum-cost perfect matching, and solving Sudoku puzzles.
  arXiv  Detail & Related papers  (2023-02-28T00:04:22Z)
• Neural Bayesian Network Understudy [13.28673601999793]
  We show that a neural network can be trained to output conditional probabilities, providing approximately the same functionality as a Bayesian Network. We propose two training strategies that allow encoding the independence relations inferred from a given causal structure into the neural network.
  arXiv  Detail & Related papers  (2022-11-15T15:56:51Z)
• Consistency of Neural Networks with Regularization [0.0]
  This paper proposes the general framework of neural networks with regularization and prove its consistency. Two types of activation functions: hyperbolic function(Tanh) and rectified linear unit(ReLU) have been taken into consideration.
  arXiv  Detail & Related papers  (2022-06-22T23:33:39Z)
• Searching for the Essence of Adversarial Perturbations [73.96215665913797]
  We show that adversarial perturbations contain human-recognizable information, which is the key conspirator responsible for a neural network's erroneous prediction. This concept of human-recognizable information allows us to explain key features related to adversarial perturbations.
  arXiv  Detail & Related papers  (2022-05-30T18:04:57Z)
• Benign Overfitting in Two-layer Convolutional Neural Networks [90.75603889605043]
  We study the benign overfitting phenomenon in training a two-layer convolutional neural network (CNN) We show that when the signal-to-noise ratio satisfies a certain condition, a two-layer CNN trained by gradient descent can achieve arbitrarily small training and test loss. On the other hand, when this condition does not hold, overfitting becomes harmful and the obtained CNN can only achieve constant level test loss.
  arXiv  Detail & Related papers  (2022-02-14T07:45:51Z)
• Analytically Tractable Hidden-States Inference in Bayesian Neural Networks [0.0]
  We show how we can leverage the tractable approximate Gaussian inference's (TAGI) capabilities to infer hidden states. One novel aspect it allows is to infer hidden states through the imposition of constraints designed to achieve specific objectives.
  arXiv  Detail & Related papers  (2021-07-08T11:11:25Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Neural Networks with Recurrent Generative Feedback [61.90658210112138]
  We instantiate this design on convolutional neural networks (CNNs) In the experiments, CNN-F shows considerably improved adversarial robustness over conventional feedforward CNNs on standard benchmarks.
  arXiv  Detail & Related papers  (2020-07-17T19:32:48Z)
• Feature Purification: How Adversarial Training Performs Robust Deep Learning [66.05472746340142]
  We show a principle that we call Feature Purification, where we show one of the causes of the existence of adversarial examples is the accumulation of certain small dense mixtures in the hidden weights during the training process of a neural network. We present both experiments on the CIFAR-10 dataset to illustrate this principle, and a theoretical result proving that for certain natural classification tasks, training a two-layer neural network with ReLU activation using randomly gradient descent indeed this principle.
  arXiv  Detail & Related papers  (2020-05-20T16:56:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.