Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

• URL: http://arxiv.org/abs/2502.18959v2
• Date: Tue, 24 Jun 2025 17:50:17 GMT
• Title: Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential
• Authors: Shijun Zhang, Hongkai Zhao, Yimin Zhong, Haomin Zhou,
• Abstract summary: We introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a novel model that creates a strong synergy between them.<n>We demonstrate that FMMNNs are highly effective and flexible in modeling high-frequency components.<n>We also analyze the optimization landscape of FMMNNs and find it to be much more favorable than that of standard fully connected neural networks.
• Score: 9.699640804685629
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The architecture of a neural network and the selection of its activation function are both fundamental to its performance. Equally vital is ensuring these two elements are well-matched, as their alignment is key to achieving effective representation and learning. In this paper, we introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), a novel model that creates a strong synergy between them. We demonstrate that FMMNNs are highly effective and flexible in modeling high-frequency components. Our theoretical results demonstrate that FMMNNs have exponential expressive power for function approximation. We also analyze the optimization landscape of FMMNNs and find it to be much more favorable than that of standard fully connected neural networks, especially when dealing with high-frequency features. In addition, we propose a scaled random initialization method for the first layer's weights in FMMNNs, which significantly speeds up training and enhances overall performance. Extensive numerical experiments support our theoretical insights, showing that FMMNNs consistently outperform traditional approaches in accuracy and efficiency across various tasks.

Related papers

• Fractional Spike Differential Equations Neural Network with Efficient Adjoint Parameters Training [63.3991315762955]
  Spiking Neural Networks (SNNs) draw inspiration from biological neurons to create realistic models for brain-like computation.<n>Most existing SNNs assume a single time constant for neuronal membrane voltage dynamics, modeled by first-order ordinary differential equations (ODEs) with Markovian characteristics.<n>We propose the Fractional SPIKE Differential Equation neural network (fspikeDE), which captures long-term dependencies in membrane voltage and spike trains through fractional-order dynamics.
  arXiv  Detail & Related papers  (2025-07-22T18:20:56Z)
• Self-cross Feature based Spiking Neural Networks for Efficient Few-shot Learning [16.156610945877986]
  We propose a few-shot learning framework based on Spiking Neural Networks (SNNs)<n>We apply the combination of temporal efficient training loss and Info Info loss to optimize the temporal spike dynamics of trains and enhance the discriminative power.
  arXiv  Detail & Related papers  (2025-05-12T16:51:08Z)
• Kolmogorov-Arnold Fourier Networks [9.17466024566914]
  Kolmogorov-Arnold-Fourier Network (KAF) integrates trainable Random Fourier Features (RFF) and a novel hybrid GELU-Fourier activation mechanism.<n>Key technical contributions include merging KAN's dual-matrix structure through matrix association properties to substantially reduce parameters.<n>Experiments demonstrate the superiority of our KAF across various domains including vision, NLP, audio processing, and differential equation-solving tasks.
  arXiv  Detail & Related papers  (2025-02-09T20:21:43Z)
• FE-UNet: Frequency Domain Enhanced U-Net with Segment Anything Capability for Versatile Image Segmentation [50.9040167152168]
  We experimentally quantify the contrast sensitivity function of CNNs and compare it with that of the human visual system.<n>We propose the Wavelet-Guided Spectral Pooling Module (WSPM) to enhance and balance image features across the frequency domain.<n>To further emulate the human visual system, we introduce the Frequency Domain Enhanced Receptive Field Block (FE-RFB)<n>We develop FE-UNet, a model that utilizes SAM2 as its backbone and incorporates Hiera-Large as a pre-trained block.
  arXiv  Detail & Related papers  (2025-02-06T07:24:34Z)
• Trainable Adaptive Activation Function Structure (TAAFS) Enhances Neural Network Force Field Performance with Only Dozens of Additional Parameters [0.0]
  Trainable Adaptive Function Activation Structure (TAAFS)<n>We introduce a method that selects distinct mathematical formulations for non-linear activations.<n>In this study, we integrate TAAFS into a variety of neural network models, resulting in observed accuracy improvements.
  arXiv  Detail & Related papers  (2024-12-19T09:06:39Z)
• Deep-Unrolling Multidimensional Harmonic Retrieval Algorithms on Neuromorphic Hardware [78.17783007774295]
  This paper explores the potential of conversion-based neuromorphic algorithms for highly accurate and energy-efficient single-snapshot multidimensional harmonic retrieval. A novel method for converting the complex-valued convolutional layers and activations into spiking neural networks (SNNs) is developed. The converted SNNs achieve almost five-fold power efficiency at moderate performance loss compared to the original CNNs.
  arXiv  Detail & Related papers  (2024-12-05T09:41:33Z)
• Neuromorphic Wireless Split Computing with Multi-Level Spikes [69.73249913506042]
  Neuromorphic computing uses spiking neural networks (SNNs) to perform inference tasks.<n> embedding a small payload within each spike exchanged between spiking neurons can enhance inference accuracy without increasing energy consumption.<n> split computing - where an SNN is partitioned across two devices - is a promising solution.<n>This paper presents the first comprehensive study of a neuromorphic wireless split computing architecture that employs multi-level SNNs.
  arXiv  Detail & Related papers  (2024-11-07T14:08:35Z)
• Multiscale fusion enhanced spiking neural network for invasive BCI neural signal decoding [13.108613110379961]
  This paper presents a novel approach utilizing a Multiscale Fusion Fusion Spiking Neural Network (MFSNN) MFSNN emulates the parallel processing and multiscale feature fusion seen in human visual perception to enable real-time, efficient, and energy-conserving neural signal decoding. MFSNN surpasses traditional artificial neural network methods, such as enhanced GRU, in both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-09-14T09:53:30Z)
• LeRF: Learning Resampling Function for Adaptive and Efficient Image Interpolation [64.34935748707673]
  Recent deep neural networks (DNNs) have made impressive progress in performance by introducing learned data priors. We propose a novel method of Learning Resampling (termed LeRF) which takes advantage of both the structural priors learned by DNNs and the locally continuous assumption. LeRF assigns spatially varying resampling functions to input image pixels and learns to predict the shapes of these resampling functions with a neural network.
  arXiv  Detail & Related papers  (2024-07-13T16:09:45Z)
• Structured and Balanced Multi-component and Multi-layer Neural Networks [9.699640804685629]
  We propose a balanced multi-component and multi-layer computation network (MMNN) MMNNs achieve a significant reduction of training parameters compared to fully connected neural networks (FCNNs) or multi-layer perceptrons (MLPs)
  arXiv  Detail & Related papers  (2024-06-30T17:00:42Z)
• Enhancing Fast Feed Forward Networks with Load Balancing and a Master Leaf Node [49.08777822540483]
  Fast feedforward networks (FFFs) exploit the observation that different regions of the input space activate distinct subsets of neurons in wide networks. We propose the incorporation of load balancing and Master Leaf techniques into the FFF architecture to improve performance and simplify the training process.
  arXiv  Detail & Related papers  (2024-05-27T05:06:24Z)
• Parallel Proportional Fusion of Spiking Quantum Neural Network for Optimizing Image Classification [10.069224006497162]
  We introduce a novel architecture termed Parallel Proportional Fusion of Quantum and Spiking Neural Networks (PPF-QSNN) The proposed PPF-QSNN outperforms both the existing spiking neural network and the serial quantum neural network across metrics such as accuracy, loss, and robustness. This study lays the groundwork for the advancement and application of quantum advantage in artificial intelligent computations.
  arXiv  Detail & Related papers  (2024-04-01T10:35:35Z)
• From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport [32.39176908225668]
  We introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for measuring the non-linearity of deep neural networks. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature.
  arXiv  Detail & Related papers  (2023-10-17T17:50:22Z)
• Artificial to Spiking Neural Networks Conversion for Scientific Machine Learning [24.799635365988905]
  We introduce a method to convert Physics-Informed Neural Networks (PINNs) to Spiking Neural Networks (SNNs) SNNs are expected to have higher energy efficiency compared to traditional Artificial Neural Networks (ANNs)
  arXiv  Detail & Related papers  (2023-08-31T00:21:27Z)
• Adaptive Frequency Filters As Efficient Global Token Mixers [100.27957692579892]
  We show that adaptive frequency filters can serve as efficient global token mixers. We take AFF token mixers as primary neural operators to build a lightweight neural network, dubbed AFFNet.
  arXiv  Detail & Related papers  (2023-07-26T07:42:28Z)
• A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks [53.31941519245432]
  Brain-inspired spiking neural networks (SNNs) have demonstrated promising capabilities in solving pattern recognition tasks. These SNNs are grounded on homogeneous neurons that utilize a uniform neural coding for information representation. In this study, we argue that SNN architectures should be holistically designed to incorporate heterogeneous coding schemes.
  arXiv  Detail & Related papers  (2023-05-26T02:52:12Z)
• Properties and Potential Applications of Random Functional-Linked Types of Neural Networks [81.56822938033119]
  Random functional-linked neural networks (RFLNNs) offer an alternative way of learning in deep structure. This paper gives some insights into the properties of RFLNNs from the viewpoints of frequency domain. We propose a method to generate a BLS network with better performance, and design an efficient algorithm for solving Poison's equation.
  arXiv  Detail & Related papers  (2023-04-03T13:25:22Z)
• SymNMF-Net for The Symmetric NMF Problem [62.44067422984995]
  We propose a neural network called SymNMF-Net for the Symmetric NMF problem. We show that the inference of each block corresponds to a single iteration of the optimization. Empirical results on real-world datasets demonstrate the superiority of our SymNMF-Net.
  arXiv  Detail & Related papers  (2022-05-26T08:17:39Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• The Spectral Bias of Polynomial Neural Networks [63.27903166253743]
  Polynomial neural networks (PNNs) have been shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a $textitspectral bias$ towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Tangent Kernel (NTK) of PNNs. We find that the $Pi$-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the
  arXiv  Detail & Related papers  (2022-02-27T23:12:43Z)
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed [27.38015169185521]
  We show theoretically that two-layer neural networks (2LNN) with only a few hidden neurons can beat the performance of kernel learning. We show how over-parametrising the neural network leads to faster convergence, but does not improve its final performance.
  arXiv  Detail & Related papers  (2021-02-23T15:10:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.