Mutual-Guided Dynamic Network for Image Fusion
- URL: http://arxiv.org/abs/2308.12538v2
- Date: Fri, 1 Sep 2023 04:51:13 GMT
- Title: Mutual-Guided Dynamic Network for Image Fusion
- Authors: Yuanshen Guan, Ruikang Xu, Mingde Yao, Lizhi Wang, Zhiwei Xiong
- Abstract summary: We propose a novel mutual-guided dynamic network (MGDN) for image fusion, which allows for effective information utilization across different locations and inputs.
Experimental results on five benchmark datasets demonstrate that our proposed method outperforms existing methods on four image fusion tasks.
- Score: 51.615598671899335
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Image fusion aims to generate a high-quality image from multiple images
captured under varying conditions. The key problem of this task is to preserve
complementary information while filtering out irrelevant information for the
fused result. However, existing methods address this problem by leveraging
static convolutional neural networks (CNNs), suffering two inherent limitations
during feature extraction, i.e., being unable to handle spatial-variant
contents and lacking guidance from multiple inputs. In this paper, we propose a
novel mutual-guided dynamic network (MGDN) for image fusion, which allows for
effective information utilization across different locations and inputs.
Specifically, we design a mutual-guided dynamic filter (MGDF) for adaptive
feature extraction, composed of a mutual-guided cross-attention (MGCA) module
and a dynamic filter predictor, where the former incorporates additional
guidance from different inputs and the latter generates spatial-variant kernels
for different locations. In addition, we introduce a parallel feature fusion
(PFF) module to effectively fuse local and global information of the extracted
features. To further reduce the redundancy among the extracted features while
simultaneously preserving their shared structural information, we devise a
novel loss function that combines the minimization of normalized mutual
information (NMI) with an estimated gradient mask. Experimental results on five
benchmark datasets demonstrate that our proposed method outperforms existing
methods on four image fusion tasks. The code and model are publicly available
at: https://github.com/Guanys-dar/MGDN.
Related papers
- A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion [41.34335755315773]
Multi-modality image fusion aims at fusing specific-modality and shared-modality information from two source images.
We propose a three-branch encoder-decoder architecture along with corresponding fusion layers as the fusion strategy.
Our method has obtained competitive results compared with state-of-the-art methods in visible/infrared image fusion and medical image fusion tasks.
arXiv Detail & Related papers (2024-06-11T09:32:40Z) - DiAD: A Diffusion-based Framework for Multi-class Anomaly Detection [55.48770333927732]
We propose a Difusion-based Anomaly Detection (DiAD) framework for multi-class anomaly detection.
It consists of a pixel-space autoencoder, a latent-space Semantic-Guided (SG) network with a connection to the stable diffusion's denoising network, and a feature-space pre-trained feature extractor.
Experiments on MVTec-AD and VisA datasets demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2023-12-11T18:38:28Z) - Accurate and lightweight dehazing via multi-receptive-field non-local
network and novel contrastive regularization [9.90146712189936]
This paper presents a multi-receptive-field non-local network (MRFNLN) for image dehazing.
It is designed as a multi-stream feature attention block (MSFAB) and cross non-local block (CNLB)
It outperforms recent state-of-the-art dehazing methods with less than 1.5 Million parameters.
arXiv Detail & Related papers (2023-09-28T14:59:16Z) - CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for
Multi-Modality Image Fusion [138.40422469153145]
We propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network.
We show that CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion.
arXiv Detail & Related papers (2022-11-26T02:40:28Z) - Semantic Labeling of High Resolution Images Using EfficientUNets and
Transformers [5.177947445379688]
We propose a new segmentation model that combines convolutional neural networks with deep transformers.
Our results demonstrate that the proposed methodology improves segmentation accuracy compared to state-of-the-art techniques.
arXiv Detail & Related papers (2022-06-20T12:03:54Z) - Weakly Aligned Feature Fusion for Multimodal Object Detection [52.15436349488198]
multimodal data often suffer from the position shift problem, i.e., the image pair is not strictly aligned.
This problem makes it difficult to fuse multimodal features and puzzles the convolutional neural network (CNN) training.
In this article, we propose a general multimodal detector named aligned region CNN (AR-CNN) to tackle the position shift problem.
arXiv Detail & Related papers (2022-04-21T02:35:23Z) - Image Fusion Transformer [75.71025138448287]
In image fusion, images obtained from different sensors are fused to generate a single image with enhanced information.
In recent years, state-of-the-art methods have adopted Convolution Neural Networks (CNNs) to encode meaningful features for image fusion.
We propose a novel Image Fusion Transformer (IFT) where we develop a transformer-based multi-scale fusion strategy.
arXiv Detail & Related papers (2021-07-19T16:42:49Z) - Inertial Sensor Data To Image Encoding For Human Action Recognition [0.0]
Convolutional Neural Networks (CNNs) are successful deep learning models in the field of computer vision.
In this paper, we use 4 types of spatial domain methods for transforming inertial sensor data to activity images.
For creating a multimodal fusion framework, we made each type of activity images multimodal by convolving with two spatial domain filters.
arXiv Detail & Related papers (2021-05-28T01:22:52Z) - Adaptive Context-Aware Multi-Modal Network for Depth Completion [107.15344488719322]
We propose to adopt the graph propagation to capture the observed spatial contexts.
We then apply the attention mechanism on the propagation, which encourages the network to model the contextual information adaptively.
Finally, we introduce the symmetric gated fusion strategy to exploit the extracted multi-modal features effectively.
Our model, named Adaptive Context-Aware Multi-Modal Network (ACMNet), achieves the state-of-the-art performance on two benchmarks.
arXiv Detail & Related papers (2020-08-25T06:00:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.