CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion

• URL: http://arxiv.org/abs/2211.14461v2
• Date: Mon, 10 Apr 2023 10:46:30 GMT
• Title: CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion
• Authors: Zixiang Zhao, Haowen Bai, Jiangshe Zhang, Yulun Zhang, Shuang Xu, Zudi Lin, Radu Timofte, Luc Van Gool
• Abstract summary: We propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. We show that CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion.
• Score: 138.40422469153145
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modality (MM) image fusion aims to render fused images that maintain the merits of different modalities, e.g., functional highlight and detailed textures. To tackle the challenge in modeling cross-modality features and decomposing desirable modality-specific and modality-shared features, we propose a novel Correlation-Driven feature Decomposition Fusion (CDDFuse) network. Firstly, CDDFuse uses Restormer blocks to extract cross-modality shallow features. We then introduce a dual-branch Transformer-CNN feature extractor with Lite Transformer (LT) blocks leveraging long-range attention to handle low-frequency global features and Invertible Neural Networks (INN) blocks focusing on extracting high-frequency local information. A correlation-driven loss is further proposed to make the low-frequency features correlated while the high-frequency features uncorrelated based on the embedded information. Then, the LT-based global fusion and INN-based local fusion layers output the fused image. Extensive experiments demonstrate that our CDDFuse achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. We also show that CDDFuse can boost the performance in downstream infrared-visible semantic segmentation and object detection in a unified benchmark. The code is available at https://github.com/Zhaozixiang1228/MMIF-CDDFuse.

Related papers

• Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond [74.96466744512992]
  The essence of image fusion is to integrate complementary information from source images. DeFusion++ produces versatile fused representations that can enhance the quality of image fusion and the effectiveness of downstream high-level vision tasks.
  arXiv  Detail & Related papers  (2024-10-16T06:28:49Z)
• DAF-Net: A Dual-Branch Feature Decomposition Fusion Network with Domain Adaptive for Infrared and Visible Image Fusion [21.64382683858586]
  Infrared and visible image fusion aims to combine complementary information from both modalities to provide a more comprehensive scene understanding. We propose a dual-branch feature decomposition fusion network (DAF-Net) with Maximum domain adaptive. By incorporating MK-MMD, the DAF-Net effectively aligns the latent feature spaces of visible and infrared images, thereby improving the quality of the fused images.
  arXiv  Detail & Related papers  (2024-09-18T02:14:08Z)
• A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusion [41.34335755315773]
  Multi-modality image fusion aims at fusing specific-modality and shared-modality information from two source images. We propose a three-branch encoder-decoder architecture along with corresponding fusion layers as the fusion strategy. Our method has obtained competitive results compared with state-of-the-art methods in visible/infrared image fusion and medical image fusion tasks.
  arXiv  Detail & Related papers  (2024-06-11T09:32:40Z)
• MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusion [4.2474907126377115]
  Multi-modality image fusion (MMIF) aims to integrate complementary information from different modalities into a single fused image. We propose a Mamba-based Dual-phase Fusion model (MambaDFuse) to extract modality-specific and modality-fused features. Our approach achieves promising fusion results in infrared-visible image fusion and medical image fusion.
  arXiv  Detail & Related papers  (2024-04-12T11:33:26Z)
• Multimodal Transformer Using Cross-Channel attention for Object Detection in Remote Sensing Images [1.662438436885552]
  Multi-modal fusion has been determined to enhance the accuracy by fusing data from multiple modalities. We propose a novel multi-modal fusion strategy for mapping relationships between different channels at the early stage. By addressing fusion in the early stage, as opposed to mid or late-stage methods, our method achieves competitive and even superior performance compared to existing techniques.
  arXiv  Detail & Related papers  (2023-10-21T00:56:11Z)
• Mutual-Guided Dynamic Network for Image Fusion [51.615598671899335]
  We propose a novel mutual-guided dynamic network (MGDN) for image fusion, which allows for effective information utilization across different locations and inputs. Experimental results on five benchmark datasets demonstrate that our proposed method outperforms existing methods on four image fusion tasks.
  arXiv  Detail & Related papers  (2023-08-24T03:50:37Z)
• DePF: A Novel Fusion Approach based on Decomposition Pooling for Infrared and Visible Images [7.11574718614606]
  A novel fusion network based on the decomposition pooling (de-pooling) manner is proposed, termed as DePF. A de-pooling based encoder is designed to extract multi-scale image and detail features of source images at the same time. The experimental results demonstrate that the proposed method exhibits superior fusion performance over the state-of-the-arts.
  arXiv  Detail & Related papers  (2023-05-27T05:47:14Z)
• An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection [59.02821429555375]
  This research focuses on the discovery and localization of hidden objects in the wild and serves unmanned systems. Through empirical analysis, infrared and visible image fusion (IVIF) enables hard-to-find objects apparent. multimodal salient object detection (SOD) accurately delineates the precise spatial location of objects within the picture.
  arXiv  Detail & Related papers  (2023-05-17T06:48:35Z)
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion [72.8898811120795]
  We propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion. Our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation.
  arXiv  Detail & Related papers  (2022-11-20T12:02:07Z)
• Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration [59.02821429555375]
  We present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion. To better fuse the registered infrared images and visible images, we present a feature Interaction Fusion Module (IFM)
  arXiv  Detail & Related papers  (2022-05-24T07:51:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.