Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion

• URL: http://arxiv.org/abs/2404.15174v1
• Date: Tue, 23 Apr 2024 16:14:20 GMT
• Title: Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion
• Authors: Yu-Jie Liang, Zihan Cao, Liang-Jian Deng, Xiao Wu,
• Abstract summary: Implicit neural representations (INR) have made significant strides in various vision-related domains. INR is prone to losing high-frequency information and is confined to the lack of global perceptual capabilities. This paper introduces a Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for MHIF task.
• Score: 12.935592400092712
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, implicit neural representations (INR) have made significant strides in various vision-related domains, providing a novel solution for Multispectral and Hyperspectral Image Fusion (MHIF) tasks. However, INR is prone to losing high-frequency information and is confined to the lack of global perceptual capabilities. To address these issues, this paper introduces a Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for MHIF task, targeting the following phenomena: The Fourier amplitudes of the HR-HSI latent code and LR-HSI are remarkably similar; however, their phases exhibit different patterns. In FeINFN, we innovatively propose a spatial and frequency implicit fusion function (Spa-Fre IFF), helping INR capture high-frequency information and expanding the receptive field. Besides, a new decoder employing a complex Gabor wavelet activation function, called Spatial-Frequency Interactive Decoder (SFID), is invented to enhance the interaction of INR features. Especially, we further theoretically prove that the Gabor wavelet activation possesses a time-frequency tightness property that favors learning the optimal bandwidths in the decoder. Experiments on two benchmark MHIF datasets verify the state-of-the-art (SOTA) performance of the proposed method, both visually and quantitatively. Also, ablation studies demonstrate the mentioned contributions. The code will be available on Anonymous GitHub (https://anonymous.4open.science/r/FeINFN-15C9/) after possible acceptance.

Related papers

• Implicit Neural Representations with Fourier Kolmogorov-Arnold Networks [4.499833362998488]
  Implicit neural representations (INRs) use neural networks to provide continuous and resolution-independent representations of complex signals. The proposed FKAN utilizes learnable activation functions modeled as Fourier series in the first layer to effectively control and learn the task-specific frequency components. Experimental results show that our proposed FKAN model outperforms three state-of-the-art baseline schemes.
  arXiv  Detail & Related papers  (2024-09-14T05:53:33Z)
• Frequency-aware Feature Fusion for Dense Image Prediction [99.85757278772262]
  We propose Frequency-Aware Feature Fusion (FreqFusion) for dense image prediction tasks. FreqFusion integrates an Adaptive Low-Pass Filter (ALPF) generator, an offset generator, and an Adaptive High-Pass Filter (AHPF) generator. Comprehensive visualization and quantitative analysis demonstrate that FreqFusion effectively improves feature consistency and sharpens object boundaries.
  arXiv  Detail & Related papers  (2024-08-23T07:30:34Z)
• FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representation [39.116375158815515]
  Implicit Neural Representation (INR) is causing a revolution in the field of signal processing. INR techniques suffer from the "frequency"-specified spectral bias and capacity-convergence gap. We propose the FINER++ framework by extending existing periodic/non-periodic activation functions to variable-periodic ones.
  arXiv  Detail & Related papers  (2024-07-28T09:24:57Z)
• NeRF-DetS: Enhancing Multi-View 3D Object Detection with Sampling-adaptive Network of Continuous NeRF-based Representation [60.47114985993196]
  NeRF-Det unifies the tasks of novel view arithmetic and 3D perception. We introduce a novel 3D perception network structure, NeRF-DetS. NeRF-DetS outperforms competitive NeRF-Det on the ScanNetV2 dataset.
  arXiv  Detail & Related papers  (2024-04-22T06:59:03Z)
• Hybrid Convolutional and Attention Network for Hyperspectral Image Denoising [54.110544509099526]
  Hyperspectral image (HSI) denoising is critical for the effective analysis and interpretation of hyperspectral data. We propose a hybrid convolution and attention network (HCANet) to enhance HSI denoising. Experimental results on mainstream HSI datasets demonstrate the rationality and effectiveness of the proposed HCANet.
  arXiv  Detail & Related papers  (2024-03-15T07:18:43Z)
• Frequency-Aware Deepfake Detection: Improving Generalizability through Frequency Space Learning [81.98675881423131]
  This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images. Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries. We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
  arXiv  Detail & Related papers  (2024-03-12T01:28:00Z)
• A Sampling Theory Perspective on Activations for Implicit Neural Representations [73.6637608397055]
  Implicit Neural Representations (INRs) have gained popularity for encoding signals as compact, differentiable entities. We conduct a comprehensive analysis of these activations from a sampling theory perspective. Our investigation reveals that sinc activations, previously unused in conjunction with INRs, are theoretically optimal for signal encoding.
  arXiv  Detail & Related papers  (2024-02-08T05:52:45Z)
• FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions [40.80112550091512]
  Implicit Neural Representation is causing a revolution in the field of signal processing. Current INR techniques suffer from a restricted capability to tune their supported frequency set. We propose variable-periodic activation functions, for which we propose FINER. We demonstrate the capabilities of FINER in the contexts of 2D image fitting, 3D signed distance field representation, and 5D neural fields radiance optimization.
  arXiv  Detail & Related papers  (2023-12-05T02:23:41Z)
• Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion [12.43436096160316]
  Multispectral and Hyperspectral Image Fusion (MHIF) is a practical task that aims to fuse a high-resolution multispectral image (HR-MSI) and a low-resolution hyperspectral image (LR-HSI) of the same scene to obtain a high-resolution hyperspectral image (HR-HSI)
  arXiv  Detail & Related papers  (2023-07-14T11:59:47Z)
• CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion [138.40422469153145]
  We propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. We show that CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2022-11-26T02:40:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.