Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential

• URL: http://arxiv.org/abs/2502.18959v1
• Date: Wed, 26 Feb 2025 09:12:52 GMT
• Title: Fourier Multi-Component and Multi-Layer Neural Networks: Unlocking High-Frequency Potential
• Authors: Shijun Zhang, Hongkai Zhao, Yimin Zhong, Haomin Zhou,
• Abstract summary: We introduce the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN)<n>We show that FMMNNs consistently achieve superior accuracy and efficiency across various tasks.
• Score: 9.699640804685629
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The two most critical ingredients of a neural network are its structure and the activation function employed, and more importantly, the proper alignment of these two that is conducive to the effective representation and learning in practice. In this work, we introduce a surprisingly effective synergy, termed the Fourier Multi-Component and Multi-Layer Neural Network (FMMNN), and demonstrate its surprising adaptability and efficiency in capturing high-frequency components. First, we theoretically establish that FMMNNs have exponential expressive power in terms of approximation capacity. Next, we analyze the optimization landscape of FMMNNs and show that it is significantly more favorable compared to fully connected neural networks. Finally, systematic and extensive numerical experiments validate our findings, demonstrating that FMMNNs consistently achieve superior accuracy and efficiency across various tasks, particularly impressive when high-frequency components are present.

Related papers

• Kolmogorov-Arnold Fourier Networks [9.17466024566914]
  Kolmogorov-Arnold-Fourier Network (KAF) integrates trainable Random Fourier Features (RFF) and a novel hybrid GELU-Fourier activation mechanism.<n>Key technical contributions include merging KAN's dual-matrix structure through matrix association properties to substantially reduce parameters.<n>Experiments demonstrate the superiority of our KAF across various domains including vision, NLP, audio processing, and differential equation-solving tasks.
  arXiv  Detail & Related papers  (2025-02-09T20:21:43Z)
• FE-UNet: Frequency Domain Enhanced U-Net with Segment Anything Capability for Versatile Image Segmentation [50.9040167152168]
  We experimentally quantify the contrast sensitivity function of CNNs and compare it with that of the human visual system.<n>We propose the Wavelet-Guided Spectral Pooling Module (WSPM) to enhance and balance image features across the frequency domain.<n>To further emulate the human visual system, we introduce the Frequency Domain Enhanced Receptive Field Block (FE-RFB)<n>We develop FE-UNet, a model that utilizes SAM2 as its backbone and incorporates Hiera-Large as a pre-trained block.
  arXiv  Detail & Related papers  (2025-02-06T07:24:34Z)
• Trainable Adaptive Activation Function Structure (TAAFS) Enhances Neural Network Force Field Performance with Only Dozens of Additional Parameters [0.0]
  Trainable Adaptive Function Activation Structure (TAAFS)<n>We introduce a method that selects distinct mathematical formulations for non-linear activations.<n>In this study, we integrate TAAFS into a variety of neural network models, resulting in observed accuracy improvements.
  arXiv  Detail & Related papers  (2024-12-19T09:06:39Z)
• Deep-Unrolling Multidimensional Harmonic Retrieval Algorithms on Neuromorphic Hardware [78.17783007774295]
  This paper explores the potential of conversion-based neuromorphic algorithms for highly accurate and energy-efficient single-snapshot multidimensional harmonic retrieval. A novel method for converting the complex-valued convolutional layers and activations into spiking neural networks (SNNs) is developed. The converted SNNs achieve almost five-fold power efficiency at moderate performance loss compared to the original CNNs.
  arXiv  Detail & Related papers  (2024-12-05T09:41:33Z)
• Neuromorphic Wireless Split Computing with Multi-Level Spikes [69.73249913506042]
  Neuromorphic computing uses spiking neural networks (SNNs) to perform inference tasks.<n> embedding a small payload within each spike exchanged between spiking neurons can enhance inference accuracy without increasing energy consumption.<n> split computing - where an SNN is partitioned across two devices - is a promising solution.<n>This paper presents the first comprehensive study of a neuromorphic wireless split computing architecture that employs multi-level SNNs.
  arXiv  Detail & Related papers  (2024-11-07T14:08:35Z)
• Structured and Balanced Multi-component and Multi-layer Neural Networks [9.699640804685629]
  We propose a balanced multi-component and multi-layer computation network (MMNN) MMNNs achieve a significant reduction of training parameters compared to fully connected neural networks (FCNNs) or multi-layer perceptrons (MLPs)
  arXiv  Detail & Related papers  (2024-06-30T17:00:42Z)
• Enhancing Fast Feed Forward Networks with Load Balancing and a Master Leaf Node [49.08777822540483]
  Fast feedforward networks (FFFs) exploit the observation that different regions of the input space activate distinct subsets of neurons in wide networks. We propose the incorporation of load balancing and Master Leaf techniques into the FFF architecture to improve performance and simplify the training process.
  arXiv  Detail & Related papers  (2024-05-27T05:06:24Z)
• Parallel Proportional Fusion of Spiking Quantum Neural Network for Optimizing Image Classification [10.069224006497162]
  We introduce a novel architecture termed Parallel Proportional Fusion of Quantum and Spiking Neural Networks (PPF-QSNN) The proposed PPF-QSNN outperforms both the existing spiking neural network and the serial quantum neural network across metrics such as accuracy, loss, and robustness. This study lays the groundwork for the advancement and application of quantum advantage in artificial intelligent computations.
  arXiv  Detail & Related papers  (2024-04-01T10:35:35Z)
• From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport [32.39176908225668]
  We introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for measuring the non-linearity of deep neural networks. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature.
  arXiv  Detail & Related papers  (2023-10-17T17:50:22Z)
• Adaptive Frequency Filters As Efficient Global Token Mixers [100.27957692579892]
  We show that adaptive frequency filters can serve as efficient global token mixers. We take AFF token mixers as primary neural operators to build a lightweight neural network, dubbed AFFNet.
  arXiv  Detail & Related papers  (2023-07-26T07:42:28Z)
• A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks [53.31941519245432]
  Brain-inspired spiking neural networks (SNNs) have demonstrated promising capabilities in solving pattern recognition tasks. These SNNs are grounded on homogeneous neurons that utilize a uniform neural coding for information representation. In this study, we argue that SNN architectures should be holistically designed to incorporate heterogeneous coding schemes.
  arXiv  Detail & Related papers  (2023-05-26T02:52:12Z)
• The Spectral Bias of Polynomial Neural Networks [63.27903166253743]
  Polynomial neural networks (PNNs) have been shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a $textitspectral bias$ towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Tangent Kernel (NTK) of PNNs. We find that the $Pi$-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the
  arXiv  Detail & Related papers  (2022-02-27T23:12:43Z)
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed [27.38015169185521]
  We show theoretically that two-layer neural networks (2LNN) with only a few hidden neurons can beat the performance of kernel learning. We show how over-parametrising the neural network leads to faster convergence, but does not improve its final performance.
  arXiv  Detail & Related papers  (2021-02-23T15:10:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.