Visible and infrared self-supervised fusion trained on a single example

• URL: http://arxiv.org/abs/2307.04100v2
• Date: Sat, 9 Mar 2024 08:16:14 GMT
• Title: Visible and infrared self-supervised fusion trained on a single example
• Authors: Nati Ofir and Jean-Christophe Nebel
• Abstract summary: Multispectral imaging is important task of image processing and computer vision. Problem of visible (RGB) to Near Infrared (NIR) image fusion has become particularly timely. Proposed approach fuses these two channels by training a Convolutional Neural Network by Self Supervised Learning (SSL) on a single example. Experiments demonstrate that the proposed approach achieves similar or better qualitative and quantitative multispectral fusion results.
• Score: 1.1188842018827656
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multispectral imaging is an important task of image processing and computer vision, which is especially relevant to applications such as dehazing or object detection. With the development of the RGBT (RGB & Thermal) sensor, the problem of visible (RGB) to Near Infrared (NIR) image fusion has become particularly timely. Indeed, while visible images see color, but suffer from noise, haze, and clouds, the NIR channel captures a clearer picture. The proposed approach fuses these two channels by training a Convolutional Neural Network by Self Supervised Learning (SSL) on a single example. For each such pair, RGB and NIR, the network is trained for seconds to deduce the final fusion. The SSL is based on the comparison of the Structure of Similarity and Edge-Preservation losses, where the labels for the SSL are the input channels themselves. This fusion preserves the relevant detail of each spectral channel without relying on a heavy training process. Experiments demonstrate that the proposed approach achieves similar or better qualitative and quantitative multispectral fusion results than other state-of-the-art methods that do not rely on heavy training and/or large datasets.

Related papers

• Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation [19.41334573257174]
  Traditional methods mostly use RGB images which are heavily affected by lighting conditions, eg, darkness. Recent studies show thermal images are robust to the night scenario as a compensating modality for segmentation. This work proposes a Residual Spatial Fusion Network (RSFNet) for RGB-T semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-17T14:28:08Z)
• Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models [54.952979335638204]
  We propose a novel method with diffusion models, termed as Dif-Fusion, to generate the distribution of the multi-channel input data. Our method is more effective than other state-of-the-art image fusion methods, especially in color fidelity.
  arXiv  Detail & Related papers  (2023-01-19T13:37:19Z)
• CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion [72.8898811120795]
  We propose a coupled contrastive learning network, dubbed CoCoNet, to realize infrared and visible image fusion. Our method achieves state-of-the-art (SOTA) performance under both subjective and objective evaluation.
  arXiv  Detail & Related papers  (2022-11-20T12:02:07Z)
• Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration [59.02821429555375]
  We present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion. To better fuse the registered infrared images and visible images, we present a feature Interaction Fusion Module (IFM)
  arXiv  Detail & Related papers  (2022-05-24T07:51:57Z)
• Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection [65.30079184700755]
  This study addresses the issue of fusing infrared and visible images that appear differently for object detection. Previous approaches discover commons underlying the two modalities and fuse upon the common space either by iterative optimization or deep networks. This paper proposes a bilevel optimization formulation for the joint problem of fusion and detection, and then unrolls to a target-aware Dual Adversarial Learning (TarDAL) network for fusion and a commonly used detection network.
  arXiv  Detail & Related papers  (2022-03-30T11:44:56Z)
• Multispectral image fusion by super pixel statistics [1.4685355149711299]
  I address the task of visible color RGB to Near-Infrared (NIR) fusion. The RGB image captures the color of the scene while the NIR captures details and sees beyond haze and clouds. The proposed method is designed to produce a fusion that contains both advantages of each spectra.
  arXiv  Detail & Related papers  (2021-12-21T16:19:10Z)
• Generation of the NIR spectral Band for Satellite Images with Convolutional Neural Networks [0.0]
  Deep neural networks allow generating artificial spectral information, such as for the image colorization problem. We study the generative adversarial network (GAN) approach in the task of the NIR band generation using just RGB channels of high-resolution satellite imagery.
  arXiv  Detail & Related papers  (2021-06-13T15:14:57Z)
• SFANet: A Spectrum-aware Feature Augmentation Network for Visible-Infrared Person Re-Identification [12.566284647658053]
  We propose a novel spectrum-aware feature augementation network named SFANet for cross-modality matching problem. Learning with grayscale-spectrum images, our model can apparently reduce modality discrepancy and detect inner structure relations. In feature-level, we improve the conventional two-stream network through balancing the number of specific and sharable convolutional blocks.
  arXiv  Detail & Related papers  (2021-02-24T08:57:32Z)
• Self-Supervised Representation Learning for RGB-D Salient Object Detection [93.17479956795862]
  We use Self-Supervised Representation Learning to design two pretext tasks: the cross-modal auto-encoder and the depth-contour estimation. Our pretext tasks require only a few and un RGB-D datasets to perform pre-training, which make the network capture rich semantic contexts. For the inherent problem of cross-modal fusion in RGB-D SOD, we propose a multi-path fusion module.
  arXiv  Detail & Related papers  (2021-01-29T09:16:06Z)
• A Dual-branch Network for Infrared and Visible Image Fusion [20.15854042473049]
  We propose a new method based on dense blocks and GANs. We directly insert the input image-visible light image in each layer of the entire network. Our experiments show that the fused images obtained by our approach achieve good score based on multiple evaluation indicators.
  arXiv  Detail & Related papers  (2021-01-24T04:18:32Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.