Evolving Neural Selection with Adaptive Regularization

• URL: http://arxiv.org/abs/2204.01662v1
• Date: Mon, 4 Apr 2022 17:19:52 GMT
• Title: Evolving Neural Selection with Adaptive Regularization
• Authors: Li Ding and Lee Spector
• Abstract summary: We show a method in which the selection of neurons in deep neural networks evolves, adapting to the difficulty of prediction. We propose the Adaptive Neural Selection (ANS) framework, which evolves to weigh neurons in a layer to form network variants. Experimental results show that the proposed method can significantly improve the performance of commonly-used neural network architectures.
• Score: 7.298440208725654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Over-parameterization is one of the inherent characteristics of modern deep neural networks, which can often be overcome by leveraging regularization methods, such as Dropout. Usually, these methods are applied globally and all the input cases are treated equally. However, given the natural variation of the input space for real-world tasks such as image recognition and natural language understanding, it is unlikely that a fixed regularization pattern will have the same effectiveness for all the input cases. In this work, we demonstrate a method in which the selection of neurons in deep neural networks evolves, adapting to the difficulty of prediction. We propose the Adaptive Neural Selection (ANS) framework, which evolves to weigh neurons in a layer to form network variants that are suitable to handle different input cases. Experimental results show that the proposed method can significantly improve the performance of commonly-used neural network architectures on standard image recognition benchmarks. Ablation studies also validate the effectiveness and contribution of each component in the proposed framework.

Related papers

• Feedback Favors the Generalization of Neural ODEs [24.342023073252395]
  We present feedback neural networks, showing that a feedback loop can flexibly correct the learned latent dynamics of neural ordinary differential equations (neural ODEs) The feedback neural network is a novel two-DOF neural network, which possesses robust performance in unseen scenarios with no loss of accuracy performance on previous tasks.
  arXiv  Detail & Related papers  (2024-10-14T08:09:45Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Function-Space Regularization in Neural Networks: A Probabilistic Perspective [51.133793272222874]
  We show that we can derive a well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training. We evaluate the utility of this regularization technique empirically and demonstrate that the proposed method leads to near-perfect semantic shift detection and highly-calibrated predictive uncertainty estimates.
  arXiv  Detail & Related papers  (2023-12-28T17:50:56Z)
• Generalizable Neural Fields as Partially Observed Neural Processes [16.202109517569145]
  We propose a new paradigm that views the large-scale training of neural representations as a part of a partially-observed neural process framework. We demonstrate that this approach outperforms both state-of-the-art gradient-based meta-learning approaches and hypernetwork approaches.
  arXiv  Detail & Related papers  (2023-09-13T01:22:16Z)
• Normalization-Equivariant Neural Networks with Application to Image Denoising [3.591122855617648]
  We propose a methodology for adapting existing neural networks so that normalization-equivariance holds by design. Our main claim is that not only ordinary convolutional layers, but also all activation functions, should be completely removed from neural networks. Experimental results in image denoising show that normalization-equivariant neural networks, in addition to their better conditioning, also provide much better generalization across noise levels.
  arXiv  Detail & Related papers  (2023-06-08T08:42:08Z)
• TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specialization [69.80141512683254]
  We introduce Tabular Neural Gradient Orthogonalization and gradient (TANGOS) TANGOS is a novel framework for regularization in the tabular setting built on latent unit attributions. We demonstrate that our approach can lead to improved out-of-sample generalization performance, outperforming other popular regularization methods.
  arXiv  Detail & Related papers  (2023-03-09T18:57:13Z)
• Neuro-Modulated Hebbian Learning for Fully Test-Time Adaptation [22.18972584098911]
  Fully test-time adaptation aims to adapt the network model based on sequential analysis of input samples during the inference stage. We take inspiration from the biological plausibility learning where the neuron responses are tuned based on a local synapse-change procedure. We design a soft Hebbian learning process which provides an unsupervised and effective mechanism for online adaptation.
  arXiv  Detail & Related papers  (2023-03-02T02:18:56Z)
• Permutation Equivariant Neural Functionals [92.0667671999604]
  This work studies the design of neural networks that can process the weights or gradients of other neural networks. We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-02-27T18:52:38Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Adaptive conversion of real-valued input into spike trains [91.3755431537592]
  This paper presents a biologically plausible method for converting real-valued input into spike trains for processing with spiking neural networks. The proposed method mimics the adaptive behaviour of retinal ganglion cells and allows input neurons to adapt their response to changes in the statistics of the input.
  arXiv  Detail & Related papers  (2021-04-12T12:33:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.