Frequency-Adaptive Pan-Sharpening with Mixture of Experts
- URL: http://arxiv.org/abs/2401.02151v1
- Date: Thu, 4 Jan 2024 08:58:25 GMT
- Title: Frequency-Adaptive Pan-Sharpening with Mixture of Experts
- Authors: Xuanhua He, Keyu Yan, Rui Li, Chengjun Xie, Jie Zhang, Man Zhou
- Abstract summary: We propose a novel Frequency Adaptive Mixture of Experts (FAME) learning framework for pan-sharpening.
Our method performs the best against other state-of-the-art ones and comprises a strong generalization ability for real-world scenes.
- Score: 22.28680499480492
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Pan-sharpening involves reconstructing missing high-frequency information in
multi-spectral images with low spatial resolution, using a higher-resolution
panchromatic image as guidance. Although the inborn connection with frequency
domain, existing pan-sharpening research has not almost investigated the
potential solution upon frequency domain. To this end, we propose a novel
Frequency Adaptive Mixture of Experts (FAME) learning framework for
pan-sharpening, which consists of three key components: the Adaptive Frequency
Separation Prediction Module, the Sub-Frequency Learning Expert Module, and the
Expert Mixture Module. In detail, the first leverages the discrete cosine
transform to perform frequency separation by predicting the frequency mask. On
the basis of generated mask, the second with low-frequency MOE and
high-frequency MOE takes account for enabling the effective low-frequency and
high-frequency information reconstruction. Followed by, the final fusion module
dynamically weights high-frequency and low-frequency MOE knowledge to adapt to
remote sensing images with significant content variations. Quantitative and
qualitative experiments over multiple datasets demonstrate that our method
performs the best against other state-of-the-art ones and comprises a strong
generalization ability for real-world scenes. Code will be made publicly at
\url{https://github.com/alexhe101/FAME-Net}.
Related papers
- Frequency-Domain Refinement with Multiscale Diffusion for Super Resolution [7.29314801047906]
We propose a novel Frequency Domain-guided multiscale Diffusion model (FDDiff)
FDDiff decomposes the high-frequency information complementing process into finer-grained steps.
We show that FDDiff outperforms prior generative methods with higher-fidelity super-resolution results.
arXiv Detail & Related papers (2024-05-16T11:58:52Z) - Frequency-Aware Deepfake Detection: Improving Generalizability through
Frequency Space Learning [81.98675881423131]
This research addresses the challenge of developing a universal deepfake detector that can effectively identify unseen deepfake images.
Existing frequency-based paradigms have relied on frequency-level artifacts introduced during the up-sampling in GAN pipelines to detect forgeries.
We introduce a novel frequency-aware approach called FreqNet, centered around frequency domain learning, specifically designed to enhance the generalizability of deepfake detectors.
arXiv Detail & Related papers (2024-03-12T01:28:00Z) - Unified Frequency-Assisted Transformer Framework for Detecting and
Grounding Multi-Modal Manipulation [109.1912721224697]
We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem.
By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts.
Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
arXiv Detail & Related papers (2023-09-18T11:06:42Z) - WaveNeRF: Wavelet-based Generalizable Neural Radiance Fields [149.2296890464997]
We design WaveNeRF, which integrates wavelet frequency decomposition into MVS and NeRF.
WaveNeRF achieves superior generalizable radiance field modeling when only given three images as input.
arXiv Detail & Related papers (2023-08-09T09:24:56Z) - Complementary Frequency-Varying Awareness Network for Open-Set
Fine-Grained Image Recognition [14.450381668547259]
Open-set image recognition is a challenging topic in computer vision.
We propose a Complementary Frequency-varying Awareness Network that could better capture both high-frequency and low-frequency information.
Based on CFAN, we propose an open-set fine-grained image recognition method, called CFAN-OSFGR.
arXiv Detail & Related papers (2023-07-14T08:15:36Z) - MSFA-Frequency-Aware Transformer for Hyperspectral Images Demosaicing [15.847332787718852]
This paper proposes a novel de-mosaicing framework, the MSFA-frequency-aware Transformer network (FDM-Net)
The advantage of Maformer is that it can leverage the MSFA information and non-local dependencies present in the data.
Our experimental results demonstrate that FDM-Net outperforms state-of-the-art methods with 6dB PSNR, and reconstructs high-fidelity details successfully.
arXiv Detail & Related papers (2023-03-23T16:27:30Z) - Masked Frequency Modeling for Self-Supervised Visual Pre-Training [102.89756957704138]
We present Masked Frequency Modeling (MFM), a unified frequency-domain-based approach for self-supervised pre-training of visual models.
MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum.
For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token.
arXiv Detail & Related papers (2022-06-15T17:58:30Z) - Adaptive Frequency Learning in Two-branch Face Forgery Detection [66.91715092251258]
We propose Adaptively learn Frequency information in the two-branch Detection framework, dubbed AFD.
We liberate our network from the fixed frequency transforms, and achieve better performance with our data- and task-dependent transform layers.
arXiv Detail & Related papers (2022-03-27T14:25:52Z) - Wavelet-Based Network For High Dynamic Range Imaging [64.66969585951207]
Existing methods, such as optical flow based and end-to-end deep learning based solutions, are error-prone either in detail restoration or ghosting artifacts removal.
In this work, we propose a novel frequency-guided end-to-end deep neural network (FNet) to conduct HDR fusion in the frequency domain, and Wavelet Transform (DWT) is used to decompose inputs into different frequency bands.
The low-frequency signals are used to avoid specific ghosting artifacts, while the high-frequency signals are used for preserving details.
arXiv Detail & Related papers (2021-08-03T12:26:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.