Neural Complexity Measures

• URL: http://arxiv.org/abs/2008.02953v2
• Date: Fri, 23 Oct 2020 07:06:55 GMT
• Title: Neural Complexity Measures
• Authors: Yoonho Lee, Juho Lee, Sung Ju Hwang, Eunho Yang, Seungjin Choi
• Abstract summary: We propose Neural Complexity (NC), a meta-learning framework for predicting generalization. Our model learns a scalar complexity measure through interactions with many heterogeneous tasks in a data-driven way.
• Score: 96.06344259626127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While various complexity measures for deep neural networks exist, specifying an appropriate measure capable of predicting and explaining generalization in deep networks has proven challenging. We propose Neural Complexity (NC), a meta-learning framework for predicting generalization. Our model learns a scalar complexity measure through interactions with many heterogeneous tasks in a data-driven way. The trained NC model can be added to the standard training loss to regularize any task learner in a standard supervised learning scenario. We contrast NC's approach against existing manually-designed complexity measures and other meta-learning models, and we validate NC's performance on multiple regression and classification tasks

Related papers

• Towards Scalable and Versatile Weight Space Learning [51.78426981947659]
  This paper introduces the SANE approach to weight-space learning. Our method extends the idea of hyper-representations towards sequential processing of subsets of neural network weights.
  arXiv  Detail & Related papers  (2024-06-14T13:12:07Z)
• Boosting Adversarial Training via Fisher-Rao Norm-based Regularization [9.975998980413301]
  We propose a novel regularization framework, called Logit-Oriented Adversarial Training (LOAT), which can mitigate the trade-off between robustness and accuracy. Our experiments demonstrate that the proposed regularization strategy can boost the performance of the prevalent adversarial training algorithms.
  arXiv  Detail & Related papers  (2024-03-26T09:22:37Z)
• Parallel Learning by Multitasking Neural Networks [1.6799377888527685]
  A modern challenge of Artificial Intelligence is learning multiple patterns at once. We show how the Multitasking Hebbian Network is naturally able to perform this complex task.
  arXiv  Detail & Related papers  (2023-08-08T07:43:31Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Generalization and Estimation Error Bounds for Model-based Neural Networks [78.88759757988761]
  We show that the generalization abilities of model-based networks for sparse recovery outperform those of regular ReLU networks. We derive practical design rules that allow to construct model-based networks with guaranteed high generalization.
  arXiv  Detail & Related papers  (2023-04-19T16:39:44Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Neural-Symbolic Recursive Machine for Systematic Generalization [113.22455566135757]
  We introduce the Neural-Symbolic Recursive Machine (NSR), whose core is a Grounded Symbol System (GSS) NSR integrates neural perception, syntactic parsing, and semantic reasoning. We evaluate NSR's efficacy across four challenging benchmarks designed to probe systematic generalization capabilities.
  arXiv  Detail & Related papers  (2022-10-04T13:27:38Z)
• Semi-Parametric Inducing Point Networks and Neural Processes [15.948270454686197]
  Semi-parametric inducing point networks (SPIN) can query the training set at inference time in a compute-efficient manner. SPIN attains linear complexity via a cross-attention mechanism between datapoints inspired by inducing point methods. In our experiments, SPIN reduces memory requirements, improves accuracy across a range of meta-learning tasks, and improves state-of-the-art performance on an important practical problem, genotype imputation.
  arXiv  Detail & Related papers  (2022-05-24T01:42:46Z)
• Can deep neural networks learn process model structure? An assessment framework and analysis [0.2580765958706854]
  We propose an evaluation scheme complemented with new fitness, precision, and generalisation metrics. We apply this framework to several process models with simple control-flow behaviour. Our results show that, even for such simplistic models, careful tuning of overfitting countermeasures is required.
  arXiv  Detail & Related papers  (2022-02-24T09:44:13Z)
• Learning for Integer-Constrained Optimization through Neural Networks with Limited Training [28.588195947764188]
  We introduce a symmetric and decomposed neural network structure, which is fully interpretable in terms of the functionality of its constituent components. By taking advantage of the underlying pattern of the integer constraint, the introduced neural network offers superior generalization performance with limited training. We show that the introduced decomposed approach can be further extended to semi-decomposed frameworks.
  arXiv  Detail & Related papers  (2020-11-10T21:17:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.