Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures

• URL: http://arxiv.org/abs/2603.16376v1
• Date: Tue, 17 Mar 2026 11:08:52 GMT
• Title: Prior-Informed Neural Network Initialization: A Spectral Approach for Function Parameterizing Architectures
• Abstract summary: We propose a prior-informed design strategy that leverages the intrinsic spectral and temporal structure of the data.<n>A principled methodology is introduced that uses the Fast Fourier Transform to extract dominant seasonal priors.<n>Experiments on synthetic and real-world benchmarks demonstrate that embedding data-driven priors significantly accelerates convergence.
• Score: 3.0098885383612104
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural network architectures designed for function parameterization, such as the Bag-of-Functions (BoF) framework, bridge the gap between the expressivity of deep learning and the interpretability of classical signal processing. However, these models are inherently sensitive to parameter initialization, as traditional data-agnostic schemes fail to capture the structural properties of the target signals, often leading to suboptimal convergence. In this work, we propose a prior-informed design strategy that leverages the intrinsic spectral and temporal structure of the data to guide both network initialization and architectural configuration. A principled methodology is introduced that uses the Fast Fourier Transform to extract dominant seasonal priors, informing model depth and initial states, and a residual-based regression approach to parameterize trend components. Crucially, this structural alignment enables a substantial reduction in encoder dimensionality without compromising reconstruction fidelity. A supporting theoretical analysis provides guidance on trend estimation under finite-sample regimes. Extensive experiments on synthetic and real-world benchmarks demonstrate that embedding data-driven priors significantly accelerates convergence, reduces performance variability across trials, and improves computational efficiency. Overall, the proposed framework enables more compact and interpretable architectures while outperforming standard initialization baselines, without altering the core training procedure.

Related papers

• From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search [9.003566823725441]
  Prediction-based approaches are widely used in neural architecture search (NAS), where a predictor estimates the performance of candidate architectures to guide selection.<n>Existing predictors are typically trained via supervised regression on limited samples, leading to overfitting and poor generalization to unseen architectures.<n>We propose a fundamentally different formulation that models performance prediction as a conditional function inference problem using a Convolutional Neural Process (ConvNP) with meta-learning capabilities.
  arXiv  Detail & Related papers  (2026-05-10T03:34:39Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• Data-Driven Variational Basis Learning Beyond Neural Networks: A Non-Neural Framework for Adaptive Basis Discovery [0.0]
  We develop a non-neural alternative that learns basis functions directly from data through variational optimization.<n>The proposed framework, termed Data Driven Variational Basis Learning (DVBL), treats basis atoms as primary optimization variables.<n>This yields a data-adaptive basis expansion that remains explicit, interpretable, and amenable to rigorous analysis.
  arXiv  Detail & Related papers  (2026-04-17T23:01:34Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning [51.505728136705564]
  We develop an analytical theory of the pretraining-fine-tuning pipeline in diagonal linear networks.<n>We find that different initialization choices place the network into four distinct fine-tuning regimes.<n>A smaller scale in earlier layers enables the network to both reuse and refine its features, leading to superior generalization.
  arXiv  Detail & Related papers  (2026-02-23T17:19:33Z)
• Deep Unfolding: Recent Developments, Theory, and Design Guidelines [99.63555420898554]
  This article provides a tutorial-style overview of deep unfolding, a framework that transforms optimization algorithms into structured, trainable ML architectures.<n>We review the foundations of optimization for inference and for learning, introduce four representative design paradigms for deep unfolding, and discuss the distinctive training schemes that arise from their iterative nature.
  arXiv  Detail & Related papers  (2025-12-03T13:16:35Z)
• Graph-Based Spectral Decomposition for Parameter Coordination in Language Model Fine-Tuning [5.69600290598441]
  The goal is to improve both fine-tuning efficiency and structural awareness during training.<n>A weighted graph is constructed, and Laplacian spectral decomposition is applied to enable frequency-domain modeling.<n>A spectral filtering mechanism is introduced during the optimization phase, enhancing the model's training stability and convergence behavior.
  arXiv  Detail & Related papers  (2025-04-28T08:42:35Z)
• Tuning-Free Structured Sparse PCA via Deep Unfolding Networks [5.931547772157972]
  We propose a new type of sparse principal component analysis (PCA) for unsupervised feature selection (UFS)<n>We use an interpretable deep unfolding network that translates iterative optimization steps into trainable neural architectures.<n>This innovation enables automatic learning of the regularization parameters, effectively bypassing the empirical tuning requirements of conventional methods.
  arXiv  Detail & Related papers  (2025-02-28T08:32:51Z)
• Generalized Factor Neural Network Model for High-dimensional Regression [50.554377879576066]
  We tackle the challenges of modeling high-dimensional data sets with latent low-dimensional structures hidden within complex, non-linear, and noisy relationships.<n>Our approach enables a seamless integration of concepts from non-parametric regression, factor models, and neural networks for high-dimensional regression.
  arXiv  Detail & Related papers  (2025-02-16T23:13:55Z)
• Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme [0.0]
  Emergence in machine learning refers to the spontaneous appearance of capabilities that arise from the scale and structure of training data.<n>We introduce a novel yet straightforward neural network initialization scheme that aims at achieving greater potential for emergence.<n>We demonstrate substantial improvements in both model accuracy and training speed, with and without batch normalization.
  arXiv  Detail & Related papers  (2024-07-26T18:56:47Z)
• End-to-End Meta-Bayesian Optimisation with Transformer Neural Processes [52.818579746354665]
  This paper proposes the first end-to-end differentiable meta-BO framework that generalises neural processes to learn acquisition functions via transformer architectures. We enable this end-to-end framework with reinforcement learning (RL) to tackle the lack of labelled acquisition data.
  arXiv  Detail & Related papers  (2023-05-25T10:58:46Z)
• DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths [45.947140164621096]
  We propose a new approach based on differential inclusions of inverse scale spaces. We show that DessiLBI unveils "winning tickets" in early epochs.
  arXiv  Detail & Related papers  (2020-07-04T04:40:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.