Deep Learning Meets Adaptive Filtering: A Stein's Unbiased Risk
Estimator Approach
- URL: http://arxiv.org/abs/2307.16708v4
- Date: Thu, 5 Oct 2023 17:34:42 GMT
- Title: Deep Learning Meets Adaptive Filtering: A Stein's Unbiased Risk
Estimator Approach
- Authors: Zahra Esmaeilbeig and Mojtaba Soltanalian
- Abstract summary: We introduce task-based deep learning frameworks, denoted as Deep RLS and Deep EASI.
These architectures transform the iterations of the original algorithms into layers of a deep neural network, enabling efficient source signal estimation.
To further enhance performance, we propose training these deep unrolled networks utilizing a surrogate loss function grounded on Stein's unbiased risk estimator (SURE)
- Score: 13.887632153924512
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This paper revisits two prominent adaptive filtering algorithms, namely
recursive least squares (RLS) and equivariant adaptive source separation
(EASI), through the lens of algorithm unrolling. Building upon the unrolling
methodology, we introduce novel task-based deep learning frameworks, denoted as
Deep RLS and Deep EASI. These architectures transform the iterations of the
original algorithms into layers of a deep neural network, enabling efficient
source signal estimation by leveraging a training process. To further enhance
performance, we propose training these deep unrolled networks utilizing a
surrogate loss function grounded on Stein's unbiased risk estimator (SURE). Our
empirical evaluations demonstrate that the Deep RLS and Deep EASI networks
outperform their underlying algorithms. Moreover, the efficacy of SURE-based
training in comparison to conventional mean squared error loss is highlighted
by numerical experiments. The unleashed potential of SURE-based training in
this paper sets a benchmark for future employment of SURE either for training
purposes or as an evaluation metric for generalization performance of neural
networks.
Related papers
- Heterogeneous Learning Rate Scheduling for Neural Architecture Search on Long-Tailed Datasets [0.0]
We propose a novel adaptive learning rate scheduling strategy tailored for the architecture parameters of DARTS.
Our approach dynamically adjusts the learning rate of the architecture parameters based on the training epoch, preventing the disruption of well-trained representations.
arXiv Detail & Related papers (2024-06-11T07:32:25Z) - Neural Active Learning Beyond Bandits [69.99592173038903]
We study both stream-based and pool-based active learning with neural network approximations.
We propose two algorithms based on the newly designed exploitation and exploration neural networks for stream-based and pool-based active learning.
arXiv Detail & Related papers (2024-04-18T21:52:14Z) - Optimal feature rescaling in machine learning based on neural networks [0.0]
An optimal rescaling of input features (OFR) is carried out by a Genetic Algorithm (GA)
The OFR reshapes the input space improving the conditioning of the gradient-based algorithm used for the training.
The approach has been tested on a FFNN modeling the outcome of a real industrial process.
arXiv Detail & Related papers (2024-02-13T21:57:31Z) - Stochastic Unrolled Federated Learning [85.6993263983062]
We introduce UnRolled Federated learning (SURF), a method that expands algorithm unrolling to federated learning.
Our proposed method tackles two challenges of this expansion, namely the need to feed whole datasets to the unrolleds and the decentralized nature of federated learning.
arXiv Detail & Related papers (2023-05-24T17:26:22Z) - The Integrated Forward-Forward Algorithm: Integrating Forward-Forward
and Shallow Backpropagation With Local Losses [0.0]
We propose an integrated method that combines the strengths of both FFA and shallow backpropagation.
We show that training neural networks with the Integrated Forward-Forward Algorithm has the potential of generating neural networks with advantageous features like robustness.
arXiv Detail & Related papers (2023-05-22T12:10:47Z) - The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF.
Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples.
In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
arXiv Detail & Related papers (2023-03-17T02:01:11Z) - Improved Algorithms for Neural Active Learning [74.89097665112621]
We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting.
We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
arXiv Detail & Related papers (2022-10-02T05:03:38Z) - Robust lEarned Shrinkage-Thresholding (REST): Robust unrolling for
sparse recover [87.28082715343896]
We consider deep neural networks for solving inverse problems that are robust to forward model mis-specifications.
We design a new robust deep neural network architecture by applying algorithm unfolding techniques to a robust version of the underlying recovery problem.
The proposed REST network is shown to outperform state-of-the-art model-based and data-driven algorithms in both compressive sensing and radar imaging problems.
arXiv Detail & Related papers (2021-10-20T06:15:45Z) - Analytically Tractable Inference in Deep Neural Networks [0.0]
Tractable Approximate Inference (TAGI) algorithm was shown to be a viable and scalable alternative to backpropagation for shallow fully-connected neural networks.
We are demonstrating how TAGI matches or exceeds the performance of backpropagation, for training classic deep neural network architectures.
arXiv Detail & Related papers (2021-03-09T14:51:34Z) - Adversarially Robust Learning via Entropic Regularization [31.6158163883893]
We propose a new family of algorithms, ATENT, for training adversarially robust deep neural networks.
Our approach achieves competitive (or better) performance in terms of robust classification accuracy.
arXiv Detail & Related papers (2020-08-27T18:54:43Z) - MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks.
The use of gradient combined nonvolutionity renders learning susceptible to novel problems.
We propose fusing neighboring layers of deeper networks that are trained with random variables.
arXiv Detail & Related papers (2020-01-28T18:25:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.