Learned Image Reasoning Prior Penetrates Deep Unfolding Network for Panchromatic and Multi-Spectral Image Fusion

• URL: http://arxiv.org/abs/2308.16083v1
• Date: Wed, 30 Aug 2023 15:15:31 GMT
• Title: Learned Image Reasoning Prior Penetrates Deep Unfolding Network for Panchromatic and Multi-Spectral Image Fusion
• Authors: Man Zhou, Jie Huang, Naishan Zheng, Chongyi Li
• Abstract summary: We propose a novel model-driven deep unfolding framework with image reasoning prior tailored for the pan-sharpening task. Our framework is motivated by the content reasoning ability of masked autoencoders with insightful designs. The uniqueness of our framework is that the holistic learning process is explicitly integrated with the inherent physical mechanism underlying the pan-sharpening task.
• Score: 45.28120834593148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The success of deep neural networks for pan-sharpening is commonly in a form of black box, lacking transparency and interpretability. To alleviate this issue, we propose a novel model-driven deep unfolding framework with image reasoning prior tailored for the pan-sharpening task. Different from existing unfolding solutions that deliver the proximal operator networks as the uncertain and vague priors, our framework is motivated by the content reasoning ability of masked autoencoders (MAE) with insightful designs. Specifically, the pre-trained MAE with spatial masking strategy, acting as intrinsic reasoning prior, is embedded into unfolding architecture. Meanwhile, the pre-trained MAE with spatial-spectral masking strategy is treated as the regularization term within loss function to constrain the spatial-spectral consistency. Such designs penetrate the image reasoning prior into deep unfolding networks while improving its interpretability and representation capability. The uniqueness of our framework is that the holistic learning process is explicitly integrated with the inherent physical mechanism underlying the pan-sharpening task. Extensive experiments on multiple satellite datasets demonstrate the superiority of our method over the existing state-of-the-art approaches. Code will be released at \url{https://manman1995.github.io/}.

Related papers

• Multi-Scale Cross-Fusion and Edge-Supervision Network for Image Splicing Localization [13.776343759641343]
  We propose a multi-scale cross-fusion and edge-supervision network for ISL. Our framework consists of three key steps: multi-scale features cross-fusion, edge mask prediction and edge-supervision localization. Our proposed method is superior to state-of-the-art schemes.
  arXiv  Detail & Related papers  (2024-12-17T03:10:04Z)
• Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening [2.874893537471256]
  Unfolding fusion methods integrate the powerful representation capabilities of deep learning with the robustness of model-based approaches. In this paper, we propose a model-based deep unfolded method for satellite image fusion. Experimental results on PRISMA, Quickbird, and WorldView2 datasets demonstrate the superior performance of our method.
  arXiv  Detail & Related papers  (2024-09-04T13:05:00Z)
• PROMPT-IML: Image Manipulation Localization with Pre-trained Foundation Models Through Prompt Tuning [35.39822183728463]
  We present a novel Prompt-IML framework for detecting tampered images. Humans tend to discern authenticity of an image based on semantic and high-frequency information. Our model can achieve better performance on eight typical fake image datasets.
  arXiv  Detail & Related papers  (2024-01-01T03:45:07Z)
• Rotation Equivariant Proximal Operator for Deep Unfolding Methods in Image Restoration [62.41329042683779]
  We propose a high-accuracy rotation equivariant proximal network that embeds rotation symmetry priors into the deep unfolding framework. This study makes efforts to suggest a high-accuracy rotation equivariant proximal network that effectively embeds rotation symmetry priors into the deep unfolding framework.
  arXiv  Detail & Related papers  (2023-12-25T11:53:06Z)
• Empowering Low-Light Image Enhancer through Customized Learnable Priors [40.83461757842304]
  We propose a paradigm for low-light image enhancement that explores the potential of customized learnable priors. Motivated by the powerful feature representation capability of Masked Autoencoder (MAE), we customize MAE-based illumination and noise priors.
  arXiv  Detail & Related papers  (2023-09-05T05:20:11Z)
• Unlocking Feature Visualization for Deeper Networks with MAgnitude Constrained Optimization [17.93878159391899]
  We describe MACO, a simple approach to generate interpretable images. Our approach yields significantly better results (both qualitatively and quantitatively) and unlocks efficient and interpretable feature visualizations for large state-of-the-art neural networks. We validate our method on a novel benchmark for comparing feature visualization methods, and release its visualizations for all classes of the ImageNet dataset.
  arXiv  Detail & Related papers  (2023-06-11T23:33:59Z)
• DELAD: Deep Landweber-guided deconvolution with Hessian and sparse prior [0.22940141855172028]
  We present a model for non-blind image deconvolution that incorporates the classic iterative method into a deep learning application. We build our network based on the iterative Landweber deconvolution algorithm, which is integrated with trainable convolutional layers to enhance the recovered image structures and details.
  arXiv  Detail & Related papers  (2022-09-30T11:15:03Z)
• PC-GANs: Progressive Compensation Generative Adversarial Networks for Pan-sharpening [50.943080184828524]
  We propose a novel two-step model for pan-sharpening that sharpens the MS image through the progressive compensation of the spatial and spectral information. The whole model is composed of triple GANs, and based on the specific architecture, a joint compensation loss function is designed to enable the triple GANs to be trained simultaneously.
  arXiv  Detail & Related papers  (2022-07-29T03:09:21Z)
• Defensive Tensorization [113.96183766922393]
  We propose tensor defensiveization, an adversarial defence technique that leverages a latent high-order factorization of the network. We empirically demonstrate the effectiveness of our approach on standard image classification benchmarks. We validate the versatility of our approach across domains and low-precision architectures by considering an audio task and binary networks.
  arXiv  Detail & Related papers  (2021-10-26T17:00:16Z)
• Deep Reparametrization of Multi-Frame Super-Resolution and Denoising [167.42453826365434]
  We propose a deep reparametrization of the maximum a posteriori formulation commonly employed in multi-frame image restoration tasks. Our approach is derived by introducing a learned error metric and a latent representation of the target image. We validate our approach through comprehensive experiments on burst denoising and burst super-resolution datasets.
  arXiv  Detail & Related papers  (2021-08-18T17:57:02Z)
• Explainability-aided Domain Generalization for Image Classification [0.0]
  We show that applying methods and architectures from the explainability literature can achieve state-of-the-art performance for the challenging task of domain generalization. We develop a set of novel algorithms including DivCAM, an approach where the network receives guidance during training via gradient based class activation maps to focus on a diverse set of discriminative features. Since these methods offer competitive performance on top of explainability, we argue that the proposed methods can be used as a tool to improve the robustness of deep neural network architectures.
  arXiv  Detail & Related papers  (2021-04-05T02:27:01Z)
• Perturbing Across the Feature Hierarchy to Improve Standard and Strict Blackbox Attack Transferability [100.91186458516941]
  We consider the blackbox transfer-based targeted adversarial attack threat model in the realm of deep neural network (DNN) image classifiers. We design a flexible attack framework that allows for multi-layer perturbations and demonstrates state-of-the-art targeted transfer performance. We analyze why the proposed methods outperform existing attack strategies and show an extension of the method in the case when limited queries to the blackbox model are allowed.
  arXiv  Detail & Related papers  (2020-04-29T16:00:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.