Frequency and Scale Perspectives of Feature Extraction
- URL: http://arxiv.org/abs/2302.12477v1
- Date: Fri, 24 Feb 2023 06:37:36 GMT
- Title: Frequency and Scale Perspectives of Feature Extraction
- Authors: Liangqi Zhang, Yihao Luo, Xiang Cao, Haibo Shen and Tianjiang Wang
- Abstract summary: We analyze the sensitivity of neural networks to frequencies and scales.
We find that neural networks have low- and medium-frequency biases but also prefer different frequency bands for different classes.
These observations lead to the hypothesis that neural networks must learn the ability to extract features at various scales and frequencies.
- Score: 5.081561820537235
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Convolutional neural networks (CNNs) have achieved superior performance but
still lack clarity about the nature and properties of feature extraction. In
this paper, by analyzing the sensitivity of neural networks to frequencies and
scales, we find that neural networks not only have low- and medium-frequency
biases but also prefer different frequency bands for different classes, and the
scale of objects influences the preferred frequency bands. These observations
lead to the hypothesis that neural networks must learn the ability to extract
features at various scales and frequencies. To corroborate this hypothesis, we
propose a network architecture based on Gaussian derivatives, which extracts
features by constructing scale space and employing partial derivatives as local
feature extraction operators to separate high-frequency information. This
manually designed method of extracting features from different scales allows
our GSSDNets to achieve comparable accuracy with vanilla networks on various
datasets.
Related papers
- Noise-Resilient Unsupervised Graph Representation Learning via Multi-Hop Feature Quality Estimation [53.91958614666386]
Unsupervised graph representation learning (UGRL) based on graph neural networks (GNNs)
We propose a novel UGRL method based on Multi-hop feature Quality Estimation (MQE)
arXiv Detail & Related papers (2024-07-29T12:24:28Z) - Model-based learning for location-to-channel mapping [0.0]
This paper presents a frugal, model-based network that separates the low frequency from the high frequency components of the target mapping function.
This yields an hypernetwork architecture where the neural network only learns low frequency sparse coefficients in a dictionary of high frequency components.
Simulation results show that the proposed neural network outperforms standard approaches on realistic synthetic data.
arXiv Detail & Related papers (2023-08-28T07:39:53Z) - A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree
Spectral Bias of Neural Networks [79.28094304325116]
Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards simpler'' functions.
We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets.
We propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies.
arXiv Detail & Related papers (2023-05-16T20:06:01Z) - Properties and Potential Applications of Random Functional-Linked Types
of Neural Networks [81.56822938033119]
Random functional-linked neural networks (RFLNNs) offer an alternative way of learning in deep structure.
This paper gives some insights into the properties of RFLNNs from the viewpoints of frequency domain.
We propose a method to generate a BLS network with better performance, and design an efficient algorithm for solving Poison's equation.
arXiv Detail & Related papers (2023-04-03T13:25:22Z) - Exploring the Approximation Capabilities of Multiplicative Neural
Networks for Smooth Functions [9.936974568429173]
We consider two classes of target functions: generalized bandlimited functions and Sobolev-Type balls.
Our results demonstrate that multiplicative neural networks can approximate these functions with significantly fewer layers and neurons.
These findings suggest that multiplicative gates can outperform standard feed-forward layers and have potential for improving neural network design.
arXiv Detail & Related papers (2023-01-11T17:57:33Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - SignalNet: A Low Resolution Sinusoid Decomposition and Estimation
Network [79.04274563889548]
We propose SignalNet, a neural network architecture that detects the number of sinusoids and estimates their parameters from quantized in-phase and quadrature samples.
We introduce a worst-case learning threshold for comparing the results of our network relative to the underlying data distributions.
In simulation, we find that our algorithm is always able to surpass the threshold for three-bit data but often cannot exceed the threshold for one-bit data.
arXiv Detail & Related papers (2021-06-10T04:21:20Z) - Performance Bounds for Neural Network Estimators: Applications in Fault
Detection [2.388501293246858]
We exploit recent results in quantifying the robustness of neural networks to construct and tune a model-based anomaly detector.
In tuning, we specifically provide upper bounds on the rate of false alarms expected under normal operation.
arXiv Detail & Related papers (2021-03-22T19:23:08Z) - Consistent feature selection for neural networks via Adaptive Group
Lasso [3.42658286826597]
We propose and establish a theoretical guarantee for the use of the adaptive group for selecting important features of neural networks.
Specifically, we show that our feature selection method is consistent for single-output feed-forward neural networks with one hidden layer and hyperbolic tangent activation function.
arXiv Detail & Related papers (2020-05-30T18:50:56Z) - Beyond Dropout: Feature Map Distortion to Regularize Deep Neural
Networks [107.77595511218429]
In this paper, we investigate the empirical Rademacher complexity related to intermediate layers of deep neural networks.
We propose a feature distortion method (Disout) for addressing the aforementioned problem.
The superiority of the proposed feature map distortion for producing deep neural network with higher testing performance is analyzed and demonstrated.
arXiv Detail & Related papers (2020-02-23T13:59:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.