Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models

• URL: http://arxiv.org/abs/2301.08072v1
• Date: Thu, 19 Jan 2023 13:37:19 GMT
• Title: Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models
• Authors: Jun Yue, Leyuan Fang, Shaobo Xia, Yue Deng, Jiayi Ma
• Abstract summary: We propose a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data. Our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity.
• Score: 54.952979335638204
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Color plays an important role in human visual perception, reflecting the spectrum of objects. However, the existing infrared and visible image fusion methods rarely explore how to handle multi-spectral/channel data directly and achieve high color fidelity. This paper addresses the above issue by proposing a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data, which increases the ability of multi-source information aggregation and the fidelity of colors. In specific, instead of converting multi-channel images into single-channel data in existing fusion methods, we create the multi-channel data distribution with a denoising network in a latent space with forward and reverse diffusion process. Then, we use the the denoising network to extract the multi-channel diffusion features with both visible and infrared information. Finally, we feed the multi-channel diffusion features to the multi-channel fusion module to directly generate the three-channel fused image. To retain the texture and intensity information, we propose multi-channel gradient loss and intensity loss. Along with the current evaluation metrics for measuring texture and intensity fidelity, we introduce a new evaluation metric to quantify color fidelity. Extensive experiments indicate that our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity.

Related papers

• TMFNet: Two-Stream Multi-Channels Fusion Networks for Color Image Operation Chain Detection [9.346492393908322]
  We propose a novel two-stream multi-channels fusion network for color image operation chain detection. The proposed method achieves state-of-the-art generalization ability while maintaining robustness to JPEG compression.
  arXiv  Detail & Related papers  (2024-09-12T02:04:26Z)
• Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas [33.334956022229846]
  We propose the Merge-Attend-Diffuse operator, which can be plugged into different types of pretrained diffusion models used in a joint diffusion setting. Specifically, we merge the diffusion paths, reprogramming self- and cross-attention to operate on the aggregated latent space. Our method maintains compatibility with the input prompt and visual quality of the generated images while increasing their semantic coherence.
  arXiv  Detail & Related papers  (2024-08-28T09:22:32Z)
• Diff-Mosaic: Augmenting Realistic Representations in Infrared Small Target Detection via Diffusion Prior [63.64088590653005]
  We propose Diff-Mosaic, a data augmentation method based on the diffusion model. We introduce an enhancement network called Pixel-Prior, which generates highly coordinated and realistic Mosaic images. In the second stage, we propose an image enhancement strategy named Diff-Prior. This strategy utilizes diffusion priors to model images in the real-world scene.
  arXiv  Detail & Related papers  (2024-06-02T06:23:05Z)
• A Dual Domain Multi-exposure Image Fusion Network based on the Spatial-Frequency Integration [57.14745782076976]
  Multi-exposure image fusion aims to generate a single high-dynamic image by integrating images with different exposures. We propose a novelty perspective on multi-exposure image fusion via the Spatial-Frequency Integration Framework, named MEF-SFI. Our method achieves visual-appealing fusion results against state-of-the-art multi-exposure image fusion approaches.
  arXiv  Detail & Related papers  (2023-12-17T04:45:15Z)
• Fusion of Infrared and Visible Images based on Spatial-Channel Attentional Mechanism [3.388001684915793]
  We present AMFusionNet, an innovative approach to infrared and visible image fusion (IVIF) By assimilating thermal details from infrared images with texture features from visible sources, our method produces images enriched with comprehensive information. Our method outperforms state-of-the-art algorithms in terms of quality and quantity.
  arXiv  Detail & Related papers  (2023-08-25T21:05:11Z)
• Visible and infrared self-supervised fusion trained on a single example [1.1188842018827656]
  Multispectral imaging is important task of image processing and computer vision. Problem of visible (RGB) to Near Infrared (NIR) image fusion has become particularly timely. Proposed approach fuses these two channels by training a Convolutional Neural Network by Self Supervised Learning (SSL) on a single example. Experiments demonstrate that the proposed approach achieves similar or better qualitative and quantitative multispectral fusion results.
  arXiv  Detail & Related papers  (2023-07-09T05:25:46Z)
• Denoising Diffusion Models for Plug-and-Play Image Restoration [135.6359475784627]
  This paper proposes DiffPIR, which integrates the traditional plug-and-play method into the diffusion sampling framework. Compared to plug-and-play IR methods that rely on discriminative Gaussian denoisers, DiffPIR is expected to inherit the generative ability of diffusion models.
  arXiv  Detail & Related papers  (2023-05-15T20:24:38Z)
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion [72.8898811120795]
  We propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion. Our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation.
  arXiv  Detail & Related papers  (2022-11-20T12:02:07Z)
• Underwater Image Enhancement via Medium Transmission-Guided Multi-Color Space Embedding [88.46682991985907]
  We present an underwater image enhancement network via medium transmission-guided multi-color space embedding, called Ucolor. Our network can effectively improve the visual quality of underwater images by exploiting multiple color spaces embedding.
  arXiv  Detail & Related papers  (2021-04-27T07:35:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.