Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System

• URL: http://arxiv.org/abs/2403.16227v1
• Date: Sun, 24 Mar 2024 16:41:50 GMT
• Title: Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System
• Authors: Jing Li, Lu Bai, Bin Yang, Chang Li, Lingfei Ma, Lixin Cui, Edwin R. Hancock,
• Abstract summary: Infrared and visible image fusion (IVF) plays an important role in intelligent transportation system (ITS) We propose a novel prior semantic guided image fusion method based on the dual-modality strategy.
• Score: 22.331591533400402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Infrared and visible image fusion (IVF) plays an important role in intelligent transportation system (ITS). The early works predominantly focus on boosting the visual appeal of the fused result, and only several recent approaches have tried to combine the high-level vision task with IVF. However, they prioritize the design of cascaded structure to seek unified suitable features and fit different tasks. Thus, they tend to typically bias toward to reconstructing raw pixels without considering the significance of semantic features. Therefore, we propose a novel prior semantic guided image fusion method based on the dual-modality strategy, improving the performance of IVF in ITS. Specifically, to explore the independent significant semantic of each modality, we first design two parallel semantic segmentation branches with a refined feature adaptive-modulation (RFaM) mechanism. RFaM can perceive the features that are semantically distinct enough in each semantic segmentation branch. Then, two pilot experiments based on the two branches are conducted to capture the significant prior semantic of two images, which then is applied to guide the fusion task in the integration of semantic segmentation branches and fusion branches. In addition, to aggregate both high-level semantics and impressive visual effects, we further investigate the frequency response of the prior semantics, and propose a multi-level representation-adaptive fusion (MRaF) module to explicitly integrate the low-frequent prior semantic with the high-frequent details. Extensive experiments on two public datasets demonstrate the superiority of our method over the state-of-the-art image fusion approaches, in terms of either the visual appeal or the high-level semantics.

Related papers

• IVGF: The Fusion-Guided Infrared and Visible General Framework [41.07925395888705]
  Infrared and visible dual-modality tasks can achieve robust performance even in extreme scenes by fusing complementary information. We propose a fusion-guided infrared and visible general framework, IVGF, which can be easily extended to many high-level vision tasks.
  arXiv  Detail & Related papers  (2024-09-02T06:38:37Z)
• From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion [66.33467192279514]
  We introduce a text-guided multi-modality image fusion method that leverages the high-level semantics from textual descriptions to integrate semantics from infrared and visible images. Our method not only produces visually superior fusion results but also achieves a higher detection mAP over existing methods, achieving state-of-the-art results.
  arXiv  Detail & Related papers  (2023-12-31T08:13:47Z)
• SSPFusion: A Semantic Structure-Preserving Approach for Infrared and Visible Image Fusion [30.55433673796615]
  Most existing learning-based infrared and visible image fusion (IVIF) methods exhibit massive redundant information in the fusion images. We propose a semantic structure-preserving approach for IVIF, namely SSPFusion. Our method is able to generate high-quality fusion images from pairs of infrared and visible images, which can boost the performance of downstream computer-vision tasks.
  arXiv  Detail & Related papers  (2023-09-26T08:13:32Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)
• Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation [66.15246197473897]
  Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. We propose a textbfMulti-textbfinteractive textbfFeature learning architecture for image fusion and textbfSegmentation.
  arXiv  Detail & Related papers  (2023-08-04T01:03:58Z)
• An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection [59.02821429555375]
  This research focuses on the discovery and localization of hidden objects in the wild and serves unmanned systems. Through empirical analysis, infrared and visible image fusion (IVIF) enables hard-to-find objects apparent. multimodal salient object detection (SOD) accurately delineates the precise spatial location of objects within the picture.
  arXiv  Detail & Related papers  (2023-05-17T06:48:35Z)
• Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond [50.556961575275345]
  We build an image fusion module to fuse complementary characteristics and cascade dual task-related modules. We develop an efficient first-order approximation to compute corresponding gradients and present dynamic weighted aggregation to balance the gradients for fusion learning.
  arXiv  Detail & Related papers  (2023-05-11T10:55:34Z)
• Breaking Free from Fusion Rule: A Fully Semantic-driven Infrared and Visible Image Fusion [51.22863068854784]
  Infrared and visible image fusion plays a vital role in the field of computer vision. Previous approaches make efforts to design various fusion rules in the loss functions. We develop a semantic-level fusion network to sufficiently utilize the semantic guidance.
  arXiv  Detail & Related papers  (2022-11-22T13:59:59Z)
• Semantic-aligned Fusion Transformer for One-shot Object Detection [18.58772037047498]
  One-shot object detection aims at detecting novel objects according to merely one given instance. Current approaches explore various feature fusions to obtain directly transferable meta-knowledge. We propose a simple but effective architecture named Semantic-aligned Fusion Transformer (SaFT) to resolve these issues.
  arXiv  Detail & Related papers  (2022-03-17T05:38:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.