Deep Fourier Up-Sampling
- URL: http://arxiv.org/abs/2210.05171v1
- Date: Tue, 11 Oct 2022 06:17:31 GMT
- Title: Deep Fourier Up-Sampling
- Authors: Man Zhou, Hu Yu, Jie Huang, Feng Zhao, Jinwei Gu, Chen Change Loy,
Deyu Meng, Chongyi Li
- Abstract summary: Up-sampling in the Fourier domain is more challenging as it does not follow such a local property.
We propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues.
- Score: 100.59885545206744
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Existing convolutional neural networks widely adopt spatial down-/up-sampling
for multi-scale modeling. However, spatial up-sampling operators (\emph{e.g.},
interpolation, transposed convolution, and un-pooling) heavily depend on local
pixel attention, incapably exploring the global dependency. In contrast, the
Fourier domain obeys the nature of global modeling according to the spectral
convolution theorem. Unlike the spatial domain that performs up-sampling with
the property of local similarity, up-sampling in the Fourier domain is more
challenging as it does not follow such a local property. In this study, we
propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve
these issues. We revisit the relationships between spatial and Fourier domains
and reveal the transform rules on the features of different resolutions in the
Fourier domain, which provide key insights for FourierUp's designs. FourierUp
as a generic operator consists of three key components: 2D discrete Fourier
transform, Fourier dimension increase rules, and 2D inverse Fourier transform,
which can be directly integrated with existing networks. Extensive experiments
across multiple computer vision tasks, including object detection, image
segmentation, image de-raining, image dehazing, and guided image
super-resolution, demonstrate the consistent performance gains obtained by
introducing our FourierUp.
Related papers
- Neural Fourier Modelling: A Highly Compact Approach to Time-Series Analysis [9.969451740838418]
We introduce Neural Fourier Modelling (NFM), a compact yet powerful solution for time-series analysis.
NFM is grounded in two key properties of the Fourier transform (FT): (i) the ability to model finite-length time series as functions in the Fourier domain, and (ii) the capacity for data manipulation within the Fourier domain.
NFM achieves state-of-the-art performance on a wide range of tasks, including challenging time-series scenarios with previously unseen sampling rates at test time.
arXiv Detail & Related papers (2024-10-07T02:39:55Z) - F2former: When Fractional Fourier Meets Deep Wiener Deconvolution and Selective Frequency Transformer for Image Deblurring [8.296475046681696]
We propose a novel approach based on the Fractional Fourier Transform (FRFT), a unified spatial-frequency representation.
We show that the performance of our proposed method is superior to other state-of-the-art (SOTA) approaches.
arXiv Detail & Related papers (2024-09-03T17:05:12Z) - FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining [71.46369218331215]
Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds.
We propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space.
arXiv Detail & Related papers (2024-05-29T18:58:59Z) - FourLLIE: Boosting Low-Light Image Enhancement by Fourier Frequency
Information [19.478293277978935]
We propose a two-stage Fourier-based Low-Light Image Enhancement (LLIE) network (FourLLIE)
In the first stage, we improve the lightness of low-light images by estimating the amplitude transform map in the Fourier space.
In the second stage, we introduce the Signal-to-Noise-Ratio (SNR) map to provide the prior for integrating the global Fourier frequency and the local spatial information.
arXiv Detail & Related papers (2023-08-06T06:14:14Z) - Fourier-Net+: Leveraging Band-Limited Representation for Efficient 3D
Medical Image Registration [62.53130123397081]
U-Net style networks are commonly utilized in unsupervised image registration to predict dense displacement fields.
We first propose Fourier-Net, which replaces the costly U-Net style expansive path with a parameter-free model-driven decoder.
We then introduce Fourier-Net+, which additionally takes the band-limited spatial representation of the images as input and further reduces the number of convolutional layers in the U-Net style network's contracting path.
arXiv Detail & Related papers (2023-07-06T13:57:12Z) - Fourier-Net: Fast Image Registration with Band-limited Deformation [16.894559169947055]
Unsupervised image registration commonly adopts U-Net style networks to predict dense displacement fields in the full-resolution spatial domain.
We propose the Fourier-Net, replacing the expansive path in a U-Net style network with a parameter-free model-driven decoder.
arXiv Detail & Related papers (2022-11-29T16:24:06Z) - Fourier Disentangled Space-Time Attention for Aerial Video Recognition [54.80846279175762]
We present an algorithm, Fourier Activity Recognition (FAR), for UAV video activity recognition.
Our formulation uses a novel Fourier object disentanglement method to innately separate out the human agent from the background.
We have evaluated our approach on multiple UAV datasets including UAV Human RGB, UAV Human Night, Drone Action, and NEC Drone.
arXiv Detail & Related papers (2022-03-21T01:24:53Z) - Seeing Implicit Neural Representations as Fourier Series [13.216389226310987]
Implicit Neural Representations (INR) use multilayer perceptrons to represent high-frequency functions in low-dimensional problem domains.
These representations achieved state-of-the-art results on tasks related to complex 3D objects and scenes.
This work analyzes the connection between the two methods and shows that a Fourier mapped perceptron is structurally like one hidden layer SIREN.
arXiv Detail & Related papers (2021-09-01T08:40:20Z) - A Fourier-based Framework for Domain Generalization [82.54650565298418]
Domain generalization aims at tackling this problem by learning transferable knowledge from multiple source domains in order to generalize to unseen target domains.
This paper introduces a novel Fourier-based perspective for domain generalization.
Experiments on three benchmarks have demonstrated that the proposed method is able to achieve state-of-the-arts performance for domain generalization.
arXiv Detail & Related papers (2021-05-24T06:50:30Z) - Learning Set Functions that are Sparse in Non-Orthogonal Fourier Bases [73.53227696624306]
We present a new family of algorithms for learning Fourier-sparse set functions.
In contrast to other work that focused on the Walsh-Hadamard transform, our novel algorithms operate with recently introduced non-orthogonal Fourier transforms.
We demonstrate effectiveness on several real-world applications.
arXiv Detail & Related papers (2020-10-01T14:31:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.