GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion

• URL: http://arxiv.org/abs/2404.15992v3
• Date: Mon, 2 Sep 2024 05:59:52 GMT
• Title: GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion
• Authors: Guosheng Lu, Zile Fang, Jiaju Tian, Haowen Huang, Yuelong Xu, Zhuolin Han, Yaoming Kang, Can Feng, Zhigang Zhao,
• Abstract summary: Infrared and visible image fusion (IVIF) aims to preserve thermal radiation information from infrared images while integrating texture details from visible images. Existing dual-discriminator generative adversarial networks (GANs) often rely on two structurally identical discriminators for learning. This paper proposes a novel GAN with a heterogeneous dual-discriminator network and an attention-based fusion strategy.
• Score: 0.1160897408844138
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Infrared and visible image fusion (IVIF) aims to preserve thermal radiation information from infrared images while integrating texture details from visible images. Thermal radiation information is mainly expressed through image intensities, while texture details are typically expressed through image gradients. However, existing dual-discriminator generative adversarial networks (GANs) often rely on two structurally identical discriminators for learning, which do not fully account for the distinct learning needs of infrared and visible image information. To this end, this paper proposes a novel GAN with a heterogeneous dual-discriminator network and an attention-based fusion strategy (GAN-HA). Specifically, recognizing the intrinsic differences between infrared and visible images, we propose, for the first time, a novel heterogeneous dual-discriminator network to simultaneously capture thermal radiation information and texture details. The two discriminators in this network are structurally different, including a salient discriminator for infrared images and a detailed discriminator for visible images. They are able to learn rich image intensity information and image gradient information, respectively. In addition, a new attention-based fusion strategy is designed in the generator to appropriately emphasize the learned information from different source images, thereby improving the information representation ability of the fusion result. In this way, the fused images generated by GAN-HA can more effectively maintain both the salience of thermal targets and the sharpness of textures. Extensive experiments on various public datasets demonstrate the superiority of GAN-HA over other state-of-the-art (SOTA) algorithms while showcasing its higher potential for practical applications.

Related papers

• DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability [75.9781362556431]
  We propose DiffDis to unify the cross-modal generative and discriminative pretraining into one single framework under the diffusion process. We show that DiffDis outperforms single-task models on both the image generation and the image-text discriminative tasks.
  arXiv  Detail & Related papers  (2023-08-18T05:03:48Z)
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion [72.8898811120795]
  We propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion. Our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation.
  arXiv  Detail & Related papers  (2022-11-20T12:02:07Z)
• An Attention-Guided and Wavelet-Constrained Generative Adversarial Network for Infrared and Visible Image Fusion [10.900528467160816]
  We propose an attention-guided and wavelet-constrained GAN for infrared and visible image fusion (AWFGAN) Specifically, we introduce the spatial attention modules (SAM) into the generator to obtain the spatial attention maps. We extend the discrimination range of visible information to the wavelet subspace, which can force the generator to restore the high-frequency details of visible images.
  arXiv  Detail & Related papers  (2022-10-20T05:01:20Z)
• PC-GANs: Progressive Compensation Generative Adversarial Networks for Pan-sharpening [50.943080184828524]
  We propose a novel two-step model for pan-sharpening that sharpens the MS image through the progressive compensation of the spatial and spectral information. The whole model is composed of triple GANs, and based on the specific architecture, a joint compensation loss function is designed to enable the triple GANs to be trained simultaneously.
  arXiv  Detail & Related papers  (2022-07-29T03:09:21Z)
• Visible and Near Infrared Image Fusion Based on Texture Information [4.718295968108302]
  A novel visible and near-infrared fusion method based on texture information is proposed to enhance unstructured environmental images. It aims at the problems of artifact, information loss and noise in traditional visible and near infrared image fusion methods. The experimental results demonstrate that the proposed algorithm can preserve the spectral characteristics and the unique information of visible and near-infrared images.
  arXiv  Detail & Related papers  (2022-07-22T09:02:17Z)
• Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection [65.30079184700755]
  This study addresses the issue of fusing infrared and visible images that appear differently for object detection. Previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks. This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network.
  arXiv  Detail & Related papers  (2022-03-30T11:44:56Z)
• Dual Contrastive Loss and Attention for GANs [82.713118646294]
  We propose a novel dual contrastive loss and show that, with this loss, discriminator learns more generalized and distinguishable representations to incentivize generation. We find attention to be still an important module for successful image generation even though it was not used in the recent state-of-the-art models. By combining the strengths of these remedies, we improve the compelling state-of-the-art Fr'echet Inception Distance (FID) by at least 17.5% on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-03-31T01:10:26Z)
• A Dual-branch Network for Infrared and Visible Image Fusion [20.15854042473049]
  We propose a new method based on dense blocks and GANs. We directly insert the input image-visible light image in each layer of the entire network. Our experiments show that the fused images obtained by our approach achieve good score based on multiple evaluation indicators.
  arXiv  Detail & Related papers  (2021-01-24T04:18:32Z)
• Interpreting Galaxy Deblender GAN from the Discriminator's Perspective [50.12901802952574]
  This research focuses on behaviors of one of the network's major components, the Discriminator, which plays a vital role but is often overlooked. We demonstrate that our method clearly reveals attention areas of the Discriminator when differentiating generated galaxy images from ground truth images.
  arXiv  Detail & Related papers  (2020-01-17T04:05:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.