Machine learning in spectral domain

• URL: http://arxiv.org/abs/2005.14436v2
• Date: Thu, 22 Oct 2020 16:13:09 GMT
• Title: Machine learning in spectral domain
• Authors: Lorenzo Giambagli, Lorenzo Buffoni, Timoteo Carletti, Walter Nocentini, Duccio Fanelli
• Abstract summary: tuning the eigenvalues correspond in fact to performing a global training of the neural network. spectral learning bound to the eigenvalues could be also employed for pre-training of deep neural networks.
• Score: 4.724825031148412
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks are usually trained in the space of the nodes, by adjusting the weights of existing links via suitable optimization protocols. We here propose a radically new approach which anchors the learning process to reciprocal space. Specifically, the training acts on the spectral domain and seeks to modify the eigenvalues and eigenvectors of transfer operators in direct space. The proposed method is ductile and can be tailored to return either linear or non-linear classifiers. Adjusting the eigenvalues, when freezing the eigenvectors entries, yields performances which are superior to those attained with standard methods {\it restricted} to a operate with an identical number of free parameters. Tuning the eigenvalues correspond in fact to performing a global training of the neural network, a procedure which promotes (resp. inhibits) collective modes on which an effective information processing relies. This is at variance with the usual approach to learning which implements instead a local modulation of the weights associated to pairwise links. Interestingly, spectral learning limited to the eigenvalues returns a distribution of the predicted weights which is close to that obtained when training the neural network in direct space, with no restrictions on the parameters to be tuned. Based on the above, it is surmised that spectral learning bound to the eigenvalues could be also employed for pre-training of deep neural networks, in conjunction with conventional machine-learning schemes. Changing the eigenvectors to a different non-orthogonal basis alters the topology of the network in direct space and thus allows to export the spectral learning strategy to other frameworks, as e.g. reservoir computing.

Related papers

• Model-Robust and Adaptive-Optimal Transfer Learning for Tackling Concept Shifts in Nonparametric Regression [7.243632426715939]
  We present a transfer learning procedure that is robust against model misspecification while adaptively attaining optimality. We derive the adaptive convergence rates of the excess risk for specifying Gaussian kernels in a prevalent class of hypothesis transfer learning algorithms.
  arXiv  Detail & Related papers  (2025-01-18T20:33:37Z)
• Tensorization of neural networks for improved privacy and interpretability [46.0920431279359]
  We present a tensorization algorithm for constructing tensor train representations of functions. We show that this approach can be used to enhance the privacy and interpretability of neural network models.
  arXiv  Detail & Related papers  (2025-01-10T19:00:06Z)
• Neural Operators with Localized Integral and Differential Kernels [77.76991758980003]
  We present a principled approach to operator learning that can capture local features under two frameworks. We prove that we obtain differential operators under an appropriate scaling of the kernel values of CNNs. To obtain local integral operators, we utilize suitable basis representations for the kernels based on discrete-continuous convolutions.
  arXiv  Detail & Related papers  (2024-02-26T18:59:31Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes [3.808063547958558]
  We study the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR. We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence.
  arXiv  Detail & Related papers  (2022-09-08T10:30:05Z)
• Convolutional Dictionary Learning by End-To-End Training of Iterative Neural Networks [3.6280929178575994]
  In this work, we construct an INN which can be used as a supervised and physics-informed online convolutional dictionary learning algorithm. We show that the proposed INN improves over two conventional model-agnostic training methods and yields competitive results also compared to a deep INN.
  arXiv  Detail & Related papers  (2022-06-09T12:15:38Z)
• On the training of sparse and dense deep neural networks: less parameters, same performance [0.0]
  We propose a variant of the spectral learning method as appeared in Giambagli et al Nat. Comm. 2021. The eigenvalues act as veritable knobs which can be freely tuned so as to (i) enhance, or alternatively silence, the contribution of the input nodes. Each spectral parameter reflects back on the whole set of inter-nodes weights, an attribute which we shall effectively exploit to yield sparse networks with stunning classification abilities.
  arXiv  Detail & Related papers  (2021-06-17T14:54:23Z)
• Learning Invariances in Neural Networks [51.20867785006147]
  We show how to parameterize a distribution over augmentations and optimize the training loss simultaneously with respect to the network parameters and augmentation parameters. We can recover the correct set and extent of invariances on image classification, regression, segmentation, and molecular property prediction from a large space of augmentations.
  arXiv  Detail & Related papers  (2020-10-22T17:18:48Z)
• Eigendecomposition-Free Training of Deep Networks for Linear Least-Square Problems [107.3868459697569]
  We introduce an eigendecomposition-free approach to training a deep network. We show that our approach is much more robust than explicit differentiation of the eigendecomposition. Our method has better convergence properties and yields state-of-the-art results.
  arXiv  Detail & Related papers  (2020-04-15T04:29:34Z)
• Understanding Self-Training for Gradual Domain Adaptation [107.37869221297687]
  We consider gradual domain adaptation, where the goal is to adapt an initial classifier trained on a source domain given only unlabeled data that shifts gradually in distribution towards a target domain. We prove the first non-vacuous upper bound on the error of self-training with gradual shifts, under settings where directly adapting to the target domain can result in unbounded error. The theoretical analysis leads to algorithmic insights, highlighting that regularization and label sharpening are essential even when we have infinite data, and suggesting that self-training works particularly well for shifts with small Wasserstein-infinity distance.
  arXiv  Detail & Related papers  (2020-02-26T08:59:40Z)
• Distance-Based Regularisation of Deep Networks for Fine-Tuning [116.71288796019809]
  We develop an algorithm that constrains a hypothesis class to a small sphere centred on the initial pre-trained weights. Empirical evaluation shows that our algorithm works well, corroborating our theoretical results.
  arXiv  Detail & Related papers  (2020-02-19T16:00:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.