On the choice of the non-trainable internal weights in random feature maps

• URL: http://arxiv.org/abs/2408.03626v1
• Date: Wed, 7 Aug 2024 08:37:23 GMT
• Title: On the choice of the non-trainable internal weights in random feature maps
• Authors: Pinak Mandal, Georg A. Gottwald,
• Abstract summary: We address the task of how to best select the internal weights for random feature maps. We show that the number of good features is the main factor controlling the forecasting skill of random feature maps. We find that random feature maps have superior forecasting capabilities whilst having several orders of magnitude lower computational cost.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The computationally cheap machine learning architecture of random feature maps can be viewed as a single-layer feedforward network in which the weights of the hidden layer are random but fixed and only the outer weights are learned via linear regression. The internal weights are typically chosen from a prescribed distribution. The choice of the internal weights significantly impacts the accuracy of random feature maps. We address here the task of how to best select the internal weights. In particular, we consider the forecasting problem whereby random feature maps are used to learn a one-step propagator map for a dynamical system. We provide a computationally cheap hit-and-run algorithm to select good internal weights which lead to good forecasting skill. We show that the number of good features is the main factor controlling the forecasting skill of random feature maps and acts as an effective feature dimension. Lastly, we compare random feature maps with single-layer feedforward neural networks in which the internal weights are now learned using gradient descent. We find that random feature maps have superior forecasting capabilities whilst having several orders of magnitude lower computational cost.

Related papers

• Learning dynamical systems with hit-and-run random feature maps [0.0]
  We show how random feature maps can be used to forecast dynamical systems with excellent forecasting skill. We introduce skip connections and construct a deep variant of random feature maps by combining several units. Our modified random feature maps provide excellent forecasting skill for both single trajectory forecasts as well as long-time estimates of statistical properties.
  arXiv  Detail & Related papers  (2025-01-11T23:19:58Z)
• Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness [86.61582747039053]
  Language model training in distributed settings is limited by the communication cost of exchanges. We extend recent work using shared randomness to perform distributed fine-tuning with low bandwidth.
  arXiv  Detail & Related papers  (2023-06-16T17:59:51Z)
• Asynchronously Trained Distributed Topographic Maps [0.0]
  We present an algorithm that uses $N$ autonomous units to generate a feature map by distributed training. Unit autonomy is achieved by sparse interaction in time & space through the combination of a distributed search, and a cascade-driven weight updating scheme.
  arXiv  Detail & Related papers  (2023-01-20T01:15:56Z)
• Generalized Differentiable RANSAC [95.95627475224231]
  $nabla$-RANSAC is a differentiable RANSAC that allows learning the entire randomized robust estimation pipeline. $nabla$-RANSAC is superior to the state-of-the-art in terms of accuracy while running at a similar speed to its less accurate alternatives.
  arXiv  Detail & Related papers  (2022-12-26T15:13:13Z)
• Adaptive Self-supervision Algorithms for Physics-informed Neural Networks [59.822151945132525]
  Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function. We study the impact of the location of the collocation points on the trainability of these models. We propose a novel adaptive collocation scheme which progressively allocates more collocation points to areas where the model is making higher errors.
  arXiv  Detail & Related papers  (2022-07-08T18:17:06Z)
• Refining neural network predictions using background knowledge [68.35246878394702]
  We show we can use logical background knowledge in learning system to compensate for a lack of labeled training data. We introduce differentiable refinement functions that find a corrected prediction close to the original prediction. This algorithm finds optimal refinements on complex SAT formulas in significantly fewer iterations and frequently finds solutions where gradient descent can not.
  arXiv  Detail & Related papers  (2022-06-10T10:17:59Z)
• Physics Informed Shallow Machine Learning for Wind Speed Prediction [66.05661813632568]
  We analyze a massive dataset of wind measured from anemometers located at 10 m height in 32 locations in Italy. We train supervised learning algorithms using the past history of wind to predict its value at a future time. We find that the optimal design as well as its performance vary with the location.
  arXiv  Detail & Related papers  (2022-04-01T14:55:10Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights? [25.102870584507244]
  We investigate the expressive power of limited depth-2 band random neural networks. A random net is a neural network where the hidden layer parameters are frozen with random bandwidth.
  arXiv  Detail & Related papers  (2020-08-19T13:26:12Z)
• Binary Stochastic Filtering: feature selection and beyond [0.0]
  This work aims at extending the neural network with ability to automatically select features by rethinking how the sparsity regularization can be used. The proposed method has demonstrated superior efficiency when compared to a few classical methods, achieved with minimal or no computational overhead.
  arXiv  Detail & Related papers  (2020-07-08T06:57:10Z)
• Train-by-Reconnect: Decoupling Locations of Weights from their Values [6.09170287691728]
  We show that untrained deep neural networks (DNNs) are different from trained ones. We propose a novel method named Lookahead Permutation (LaPerm) to train DNNs by reconnecting the weights. When the initial weights share a single value, our method finds weight neural network with far better-than-chance accuracy.
  arXiv  Detail & Related papers  (2020-03-05T12:40:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.