MISFIT-V: Misaligned Image Synthesis and Fusion using Information from
Thermal and Visual
- URL: http://arxiv.org/abs/2309.13216v1
- Date: Fri, 22 Sep 2023 23:41:24 GMT
- Title: MISFIT-V: Misaligned Image Synthesis and Fusion using Information from
Thermal and Visual
- Authors: Aadhar Chauhan, Isaac Remy, Danny Broyles, and Karen Leung
- Abstract summary: This work presents Misaligned Image Synthesis and Fusion using Information from Thermal and Visual (MISFIT-V)
It is a novel two-pronged unsupervised deep learning approach that utilizes a Generative Adversarial Network (GAN) and a cross-attention mechanism to capture the most relevant features from each modality.
Experimental results show MISFIT-V offers enhanced robustness against misalignment and poor lighting/thermal environmental conditions.
- Score: 2.812395851874055
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Detecting humans from airborne visual and thermal imagery is a fundamental
challenge for Wilderness Search-and-Rescue (WiSAR) teams, who must perform this
function accurately in the face of immense pressure. The ability to fuse these
two sensor modalities can potentially reduce the cognitive load on human
operators and/or improve the effectiveness of computer vision object detection
models. However, the fusion task is particularly challenging in the context of
WiSAR due to hardware limitations and extreme environmental factors. This work
presents Misaligned Image Synthesis and Fusion using Information from Thermal
and Visual (MISFIT-V), a novel two-pronged unsupervised deep learning approach
that utilizes a Generative Adversarial Network (GAN) and a cross-attention
mechanism to capture the most relevant features from each modality.
Experimental results show MISFIT-V offers enhanced robustness against
misalignment and poor lighting/thermal environmental conditions compared to
existing visual-thermal image fusion methods.
Related papers
- VIFNet: An End-to-end Visible-Infrared Fusion Network for Image Dehazing [13.777195433138179]
This study aims to design a visible-infrared fusion network for image dehazing.
In particular, we propose a multi-scale Deep Structure Feature Extraction (DSFE) module to restore more spatial and marginal information.
To validate this, we construct a visible-infrared multimodal dataset called AirSim-VID based on the AirSim simulation platform.
arXiv Detail & Related papers (2024-04-11T14:31:11Z) - Hybrid-Supervised Dual-Search: Leveraging Automatic Learning for
Loss-free Multi-Exposure Image Fusion [60.221404321514086]
Multi-exposure image fusion (MEF) has emerged as a prominent solution to address the limitations of digital imaging in representing varied exposure levels.
This paper presents a Hybrid-Supervised Dual-Search approach for MEF, dubbed HSDS-MEF, which introduces a bi-level optimization search scheme for automatic design of both network structures and loss functions.
arXiv Detail & Related papers (2023-09-03T08:07:26Z) - An Interactively Reinforced Paradigm for Joint Infrared-Visible Image
Fusion and Saliency Object Detection [59.02821429555375]
This research focuses on the discovery and localization of hidden objects in the wild and serves unmanned systems.
Through empirical analysis, infrared and visible image fusion (IVIF) enables hard-to-find objects apparent.
multimodal salient object detection (SOD) accurately delineates the precise spatial location of objects within the picture.
arXiv Detail & Related papers (2023-05-17T06:48:35Z) - Critical heat flux diagnosis using conditional generative adversarial
networks [0.0]
The critical heat flux (CHF) is an essential safety boundary in boiling heat transfer processes employed in high heat flux thermal-hydraulic systems.
This study presents a data-driven, image-to-image translation method for reconstructing thermal data of a boiling system at CHF.
arXiv Detail & Related papers (2023-05-04T07:53:04Z) - CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature
Ensemble for Multi-modality Image Fusion [72.8898811120795]
We propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion.
Our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation.
arXiv Detail & Related papers (2022-11-20T12:02:07Z) - Target-aware Dual Adversarial Learning and a Multi-scenario
Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection [65.30079184700755]
This study addresses the issue of fusing infrared and visible images that appear differently for object detection.
Previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks.
This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network.
arXiv Detail & Related papers (2022-03-30T11:44:56Z) - Does Thermal data make the detection systems more reliable? [1.2891210250935146]
We propose a comprehensive detection system based on a multimodal-collaborative framework.
This framework learns from both RGB (from visual cameras) and thermal (from Infrared cameras) data.
Our empirical results show that while the improvement in accuracy is nominal, the value lies in challenging and extremely difficult edge cases.
arXiv Detail & Related papers (2021-11-09T15:04:34Z) - Simultaneous Face Hallucination and Translation for Thermal to Visible
Face Verification using Axial-GAN [74.22129648654783]
We introduce the task of thermal-to-visible face verification from low-resolution thermal images.
We propose Axial-Generative Adversarial Network (Axial-GAN) to synthesize high-resolution visible images for matching.
arXiv Detail & Related papers (2021-04-13T22:34:28Z) - SSTN: Self-Supervised Domain Adaptation Thermal Object Detection for
Autonomous Driving [6.810856082577402]
We have proposed a deep neural network Self Supervised Thermal Network (SSTN) for learning the feature embedding to maximize the information between visible and infrared spectrum domain by contrastive learning.
The proposed method is extensively evaluated on the two publicly available datasets: the FLIR-ADAS dataset and the KAIST Multi-Spectral dataset.
arXiv Detail & Related papers (2021-03-04T16:42:49Z) - Exploring Thermal Images for Object Detection in Underexposure Regions
for Autonomous Driving [67.69430435482127]
Underexposure regions are vital to construct a complete perception of the surroundings for safe autonomous driving.
The availability of thermal cameras has provided an essential alternate to explore regions where other optical sensors lack in capturing interpretable signals.
This work proposes a domain adaptation framework which employs a style transfer technique for transfer learning from visible spectrum images to thermal images.
arXiv Detail & Related papers (2020-06-01T09:59:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.