Target noise: A pre-training based neural network initialization for efficient high resolution learning

• URL: http://arxiv.org/abs/2602.06585v1
• Date: Fri, 06 Feb 2026 10:26:12 GMT
• Title: Target noise: A pre-training based neural network initialization for efficient high resolution learning
• Authors: Shaowen Wang, Tariq Alkhalifah,
• Abstract summary: We propose a simple, yet effective, neural network pre-training strategy based on self-supervised pre-training using random noise as the target.<n>We show that this noise-driven pre-training significantly improves convergence speed in subsequent tasks, without requiring additional data or changes to the network architecture.
• Score: 4.186531258193163
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Weight initialization plays a crucial role in the optimization behavior and convergence efficiency of neural networks. Most existing initialization methods, such as Xavier and Kaiming initializations, rely on random sampling and do not exploit information from the optimization process itself. We propose a simple, yet effective, initialization strategy based on self-supervised pre-training using random noise as the target. Instead of directly training the network from random weights, we first pre-train it to fit random noise, which leads to a structured and non-random parameter configuration. We show that this noise-driven pre-training significantly improves convergence speed in subsequent tasks, without requiring additional data or changes to the network architecture. The proposed method is particularly effective for implicit neural representations (INRs) and Deep Image Prior (DIP)-style networks, which are known to exhibit a strong low-frequency bias during optimization. After noise-based pre-training, the network is able to capture high-frequency components much earlier in training, leading to faster and more stable convergence. Although random noise contains no semantic information, it serves as an effective self-supervised signal (considering its white spectrum nature) for shaping the initialization of neural networks. Overall, this work demonstrates that noise-based pre-training offers a lightweight and general alternative to traditional random initialization, enabling more efficient optimization of deep neural networks.

Related papers

• Neural network initialization with nonlinear characteristics and information on spectral bias [0.0]
  Initialization of neural network parameters, such as weights and biases, has a crucial impact on learning performance.<n>We propose a framework that adjusts the scale factors in the SWIM algorithm to capture low-frequency components in the early-stage hidden layers.
  arXiv  Detail & Related papers  (2025-11-04T04:15:32Z)
• NeRF-based CBCT Reconstruction needs Normalization and Initialization [53.58395475423445]
  NeRF-based methods suffer from a local-global training mismatch between their two key components: the hash encoder and the neural network.<n>We introduce a Normalized Hash, which enhances feature consistency and mitigates the mismatch.<n>The neural network exhibits improved stability during early training, enabling faster convergence and enhanced reconstruction performance.
  arXiv  Detail & Related papers  (2025-06-24T16:01:45Z)
• Unrolled denoising networks provably learn optimal Bayesian inference [54.79172096306631]
  We prove the first rigorous learning guarantees for neural networks based on unrolling approximate message passing (AMP) For compressed sensing, we prove that when trained on data drawn from a product prior, the layers of the network converge to the same denoisers used in Bayes AMP.
  arXiv  Detail & Related papers  (2024-09-19T17:56:16Z)
• Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization [49.06421851486415]
  Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. We propose Exact Orthogonal Initialization (EOI), a novel sparse Orthogonal Initialization scheme based on random Givens rotations. Our method enables training highly sparse 1000-layer and CNN networks without residual connections or normalization techniques.
  arXiv  Detail & Related papers  (2024-06-03T19:44:47Z)
• Pretraining with Random Noise for Fast and Robust Learning without Weight Transport [6.916179672407521]
  We show that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport.<n> Sequential training with both random noise and data brings weights closer to synaptic feedback than training solely with data.<n>This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization loss during subsequent training.
  arXiv  Detail & Related papers  (2024-05-27T00:12:51Z)
• Robust Learning of Parsimonious Deep Neural Networks [0.0]
  We propose a simultaneous learning and pruning algorithm capable of identifying and eliminating irrelevant structures in a neural network. We derive a novel hyper-prior distribution over the prior parameters that is crucial for their optimal selection. We evaluate the proposed algorithm on the MNIST data set and commonly used fully connected and convolutional LeNet architectures.
  arXiv  Detail & Related papers  (2022-05-10T03:38:55Z)
• A Bregman Learning Framework for Sparse Neural Networks [1.7778609937758323]
  We propose a learning framework based on Bregman iterations to train sparse neural networks. We derive a baseline algorithm called LinBreg, an accelerated version using momentum, and AdaBreg, which is a Bregmanized generalization of the Adam algorithm.
  arXiv  Detail & Related papers  (2021-05-10T12:56:01Z)
• Data-driven Weight Initialization with Sylvester Solvers [72.11163104763071]
  We propose a data-driven scheme to initialize the parameters of a deep neural network. We show that our proposed method is especially effective in few-shot and fine-tuning settings.
  arXiv  Detail & Related papers  (2021-05-02T07:33:16Z)
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment [84.57874289554839]
  Training deep neural networks on large-scale datasets requires significant hardware resources. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. We propose a neuro-biologically-plausible alternative to backprop that can be used to train deep networks.
  arXiv  Detail & Related papers  (2020-02-10T16:20:02Z)
• MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
  Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. The use of gradient combined nonvolutionity renders learning susceptible to novel problems. We propose fusing neighboring layers of deeper networks that are trained with random variables.
  arXiv  Detail & Related papers  (2020-01-28T18:25:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.