FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining

• URL: http://arxiv.org/abs/2405.19450v2
• Date: Wed, 7 Aug 2024 17:30:16 GMT
• Title: FourierMamba: Fourier Learning Integration with State Space Models for Image Deraining
• Authors: Dong Li, Yidi Liu, Xueyang Fu, Senyan Xu, Zheng-Jun Zha,
• Abstract summary: Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds. We propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space.
• Score: 71.46369218331215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image deraining aims to remove rain streaks from rainy images and restore clear backgrounds. Currently, some research that employs the Fourier transform has proved to be effective for image deraining, due to it acting as an effective frequency prior for capturing rain streaks. However, despite there exists dependency of low frequency and high frequency in images, these Fourier-based methods rarely exploit the correlation of different frequencies for conjuncting their learning procedures, limiting the full utilization of frequency information for image deraining. Alternatively, the recently emerged Mamba technique depicts its effectiveness and efficiency for modeling correlation in various domains (e.g., spatial, temporal), and we argue that introducing Mamba into its unexplored Fourier spaces to correlate different frequencies would help improve image deraining. This motivates us to propose a new framework termed FourierMamba, which performs image deraining with Mamba in the Fourier space. Owning to the unique arrangement of frequency orders in Fourier space, the core of FourierMamba lies in the scanning encoding of different frequencies, where the low-high frequency order formats exhibit differently in the spatial dimension (unarranged in axis) and channel dimension (arranged in axis). Therefore, we design FourierMamba that correlates Fourier space information in the spatial and channel dimensions with distinct designs. Specifically, in the spatial dimension Fourier space, we introduce the zigzag coding to scan the frequencies to rearrange the orders from low to high frequencies, thereby orderly correlating the connections between frequencies; in the channel dimension Fourier space with arranged orders of frequencies in axis, we can directly use Mamba to perform frequency correlation and improve the channel information representation.

Related papers

• Wavelet-based Mamba with Fourier Adjustment for Low-light Image Enhancement [26.13172849144202]
  We propose a novel Wavelet-based Mamba with Fourier Adjustment model called WalMaFa. WMB is adopted in the Decoder and FFAB is adopted in the Latent-Decoder structure. Experiments demonstrate that our proposed WalMaFa achieves state-of-the-art performance with fewer computational resources and faster speed.
  arXiv  Detail & Related papers  (2024-10-27T02:48:28Z)
• F2former: When Fractional Fourier Meets Deep Wiener Deconvolution and Selective Frequency Transformer for Image Deblurring [8.296475046681696]
  We propose a novel approach based on the Fractional Fourier Transform (FRFT), a unified spatial-frequency representation. We show that the performance of our proposed method is superior to other state-of-the-art (SOTA) approaches.
  arXiv  Detail & Related papers  (2024-09-03T17:05:12Z)
• FreqMamba: Viewing Mamba from a Frequency Perspective for Image Deraining [1.6793052475826054]
  Images corrupted by rain streaks often lose vital frequency information for perception, and image deraining aims to solve this issue. Recent studies have witnessed the effectiveness and efficiency of Mamba for perceiving global and local information. We propose FreqMamba, an effective and efficient paradigm that leverages the complementary between Mamba and frequency analysis for image deraining.
  arXiv  Detail & Related papers  (2024-04-15T06:02:31Z)
• Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning [81.98675881423131]
  This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
  arXiv  Detail & Related papers  (2024-03-12T01:28:00Z)
• FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization [67.47895278233717]
  We develop a progressive frequency regularization technique to tackle the over-reconstruction issue within the frequency space. FreGS achieves superior novel view synthesis and outperforms the state-of-the-art consistently.
  arXiv  Detail & Related papers  (2024-03-11T17:00:27Z)
• Neural Fourier Filter Bank [18.52741992605852]
  We present a novel method to provide efficient and highly detailed reconstructions. Inspired by wavelets, we learn a neural field that decompose the signal both spatially and frequency-wise.
  arXiv  Detail & Related papers  (2022-12-04T03:45:08Z)
• Deep Fourier Up-Sampling [100.59885545206744]
  Up-sampling in the Fourier domain is more challenging as it does not follow such a local property. We propose a theoretically sound Deep Fourier Up-Sampling (FourierUp) to solve these issues.
  arXiv  Detail & Related papers  (2022-10-11T06:17:31Z)
• Fourier Disentangled Space-Time Attention for Aerial Video Recognition [54.80846279175762]
  We present an algorithm, Fourier Activity Recognition (FAR), for UAV video activity recognition. Our formulation uses a novel Fourier object disentanglement method to innately separate out the human agent from the background. We have evaluated our approach on multiple UAV datasets including UAV Human RGB, UAV Human Night, Drone Action, and NEC Drone.
  arXiv  Detail & Related papers  (2022-03-21T01:24:53Z)
• WaveFill: A Wavelet-based Generation Network for Image Inpainting [57.012173791320855]
  WaveFill is a wavelet-based inpainting network that decomposes images into multiple frequency bands. WaveFill decomposes images by using discrete wavelet transform (DWT) that preserves spatial information naturally. It applies L1 reconstruction loss to the low-frequency bands and adversarial loss to high-frequency bands, hence effectively mitigate inter-frequency conflicts.
  arXiv  Detail & Related papers  (2021-07-23T04:44:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.