Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regression

• URL: http://arxiv.org/abs/2410.06399v2
• Date: Wed, 27 Nov 2024 20:24:36 GMT
• Title: Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regression
• Authors: Aku Kammonen, Anamika Pandey, Erik von Schwerin, Raúl Tempone,
• Abstract summary: We present an enhanced adaptive random Fourier features (ARFF) training algorithm for shallow neural networks. This method uses a particle filter-type resampling technique to stabilize the training process and reduce the sensitivity to parameter choices.
• Score: 0.8947831206263182
• License:
• Abstract: This paper presents an enhanced adaptive random Fourier features (ARFF) training algorithm for shallow neural networks, building upon the work introduced in "Adaptive Random Fourier Features with Metropolis Sampling", Kammonen et al., \emph{Foundations of Data Science}, 2(3):309--332, 2020. This improved method uses a particle filter-type resampling technique to stabilize the training process and reduce the sensitivity to parameter choices. The Metropolis test can also be omitted when resampling is used, reducing the number of hyperparameters by one and reducing the computational cost per iteration compared to the ARFF method. We present comprehensive numerical experiments demonstrating the efficacy of the proposed algorithm in function regression tasks as a stand-alone method and as a pretraining step before gradient-based optimization, using the Adam optimizer. Furthermore, we apply the proposed algorithm to a simple image regression problem, illustrating its utility in sampling frequencies for the random Fourier features (RFF) layer of coordinate-based multilayer perceptrons. In this context, we use the proposed algorithm to sample the parameters of the RFF layer in an automated manner.

Related papers

• Optimized Sampling for Non-Line-of-Sight Imaging Using Modified Fast Fourier Transforms [6.866110149269]
  Non-line-of-Sight (NLOS) imaging systems collect light at a diffuse relay surface and input this measurement into computational algorithms that output a 3D reconstruction. These algorithms utilize the Fast Fourier Transform (FFT) to accelerate the reconstruction process but require both input and output to be sampled spatially with uniform grids. In this work, we demonstrate that existing NLOS imaging setups typically oversample the relay surface spatially, explaining why the measurement can be compressed without sacrificing reconstruction quality.
  arXiv  Detail & Related papers  (2025-01-09T13:52:30Z)
• Adaptive Selection of Sampling-Reconstruction in Fourier Compressed Sensing [13.775902519100075]
  Compressed sensing (CS) has emerged to overcome the inefficiency of Nyquist sampling. Deep learning-based reconstruction has been a promising alternative to optimization-based reconstruction.
  arXiv  Detail & Related papers  (2024-09-18T06:51:29Z)
• Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction [0.0]
  This paper aims to enhance the use of the Frank-Wolfe (FW) algorithm for training deep neural networks. Similar to any-based algorithm, FW suffers from high computational memory costs when computing for DNNs.
  arXiv  Detail & Related papers  (2024-03-19T07:25:36Z)
• Data-Driven Filter Design in FBP: Transforming CT Reconstruction with Trainable Fourier Series [3.6508148866314163]
  We introduce a trainable filter for computed tomography (CT) reconstruction within the filtered backprojection (FBP) framework. This method overcomes the limitation in noise reduction by optimizing Fourier series coefficients to construct the filter. Our filter can be easily integrated into existing CT reconstruction models, making it an adaptable tool for a wide range of practical applications.
  arXiv  Detail & Related papers  (2024-01-29T10:47:37Z)
• On Optimal Sampling for Learning SDF Using MLPs Equipped with Positional Encoding [76.78321723602636]
  We devise a tool to determine the appropriate sampling rate for learning an accurate neural implicit field without undesirable side effects. It is observed that a PE-equipped has an intrinsic frequency much higher than the highest frequency component in the PE layer. We empirically show in the setting of SDF fitting, this recommended sampling rate is sufficient to secure accurate fitting results.
  arXiv  Detail & Related papers  (2024-01-02T10:51:52Z)
• Neural Fields with Thermal Activations for Arbitrary-Scale Super-Resolution [56.089473862929886]
  We present a novel way to design neural fields such that points can be queried with an adaptive Gaussian PSF. With its theoretically guaranteed anti-aliasing, our method sets a new state of the art for arbitrary-scale single image super-resolution.
  arXiv  Detail & Related papers  (2023-11-29T14:01:28Z)
• An Optimization-based Deep Equilibrium Model for Hyperspectral Image Deconvolution with Convergence Guarantees [71.57324258813675]
  We propose a novel methodology for addressing the hyperspectral image deconvolution problem. A new optimization problem is formulated, leveraging a learnable regularizer in the form of a neural network. The derived iterative solver is then expressed as a fixed-point calculation problem within the Deep Equilibrium framework.
  arXiv  Detail & Related papers  (2023-06-10T08:25:16Z)
• Low-rank extended Kalman filtering for online learning of neural networks from streaming data [71.97861600347959]
  We propose an efficient online approximate Bayesian inference algorithm for estimating the parameters of a nonlinear function from a potentially non-stationary data stream. The method is based on the extended Kalman filter (EKF), but uses a novel low-rank plus diagonal decomposition of the posterior matrix. In contrast to methods based on variational inference, our method is fully deterministic, and does not require step-size tuning.
  arXiv  Detail & Related papers  (2023-05-31T03:48:49Z)
• Transform Once: Efficient Operator Learning in Frequency Domain [69.74509540521397]
  We study deep neural networks designed to harness the structure in frequency domain for efficient learning of long-range correlations in space or time. This work introduces a blueprint for frequency domain learning through a single transform: transform once (T1)
  arXiv  Detail & Related papers  (2022-11-26T01:56:05Z)
• Supervised Learning and the Finite-Temperature String Method for Computing Committor Functions and Reaction Rates [0.0]
  A central object in the computational studies of rare events is the committor function. We show additional modifications are needed to improve the accuracy of the algorithm.
  arXiv  Detail & Related papers  (2021-07-28T17:44:00Z)
• Learning Sampling Policy for Faster Derivative Free Optimization [100.27518340593284]
  We propose a new reinforcement learning based ZO algorithm (ZO-RL) with learning the sampling policy for generating the perturbations in ZO optimization instead of using random sampling. Our results show that our ZO-RL algorithm can effectively reduce the variances of ZO gradient by learning a sampling policy, and converge faster than existing ZO algorithms in different scenarios.
  arXiv  Detail & Related papers  (2021-04-09T14:50:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.