Towards RGB-NIR Cross-modality Image Registration and Beyond

• URL: http://arxiv.org/abs/2405.19914v1
• Date: Thu, 30 May 2024 10:25:50 GMT
• Title: Towards RGB-NIR Cross-modality Image Registration and Beyond
• Authors: Huadong Li, Shichao Dong, Jin Wang, Rong Fu, Minhao Jing, Jiajun Liang, Haoqiang Fan, Renhe Ji,
• Abstract summary: This paper focuses on the area of RGB(visible)-NIR(near-infrared) cross-modality image registration. We first present the RGB-NIR Image Registration (RGB-NIR-IRegis) benchmark, which, for the first time, enables fair and comprehensive evaluations. We then design several metrics to reveal the toxic impact of inconsistent local features between visible and infrared images on the model performance.
• Score: 21.475871648254564
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper focuses on the area of RGB(visible)-NIR(near-infrared) cross-modality image registration, which is crucial for many downstream vision tasks to fully leverage the complementary information present in visible and infrared images. In this field, researchers face two primary challenges - the absence of a correctly-annotated benchmark with viewpoint variations for evaluating RGB-NIR cross-modality registration methods and the problem of inconsistent local features caused by the appearance discrepancy between RGB-NIR cross-modality images. To address these challenges, we first present the RGB-NIR Image Registration (RGB-NIR-IRegis) benchmark, which, for the first time, enables fair and comprehensive evaluations for the task of RGB-NIR cross-modality image registration. Evaluations of previous methods highlight the significant challenges posed by our RGB-NIR-IRegis benchmark, especially on RGB-NIR image pairs with viewpoint variations. To analyze the causes of the unsatisfying performance, we then design several metrics to reveal the toxic impact of inconsistent local features between visible and infrared images on the model performance. This further motivates us to develop a baseline method named Semantic Guidance Transformer (SGFormer), which utilizes high-level semantic guidance to mitigate the negative impact of local inconsistent features. Despite the simplicity of our motivation, extensive experimental results show the effectiveness of our method.

Related papers

• Pix2Next: Leveraging Vision Foundation Models for RGB to NIR Image Translation [0.536022165180739]
  We propose a novel image-to-image translation framework, Pix2Next, to generate high-quality Near-Infrared (NIR) images from RGB inputs. A multi-scale PatchGAN discriminator ensures realistic image generation at various detail levels, while carefully designed loss functions couple global context understanding with local feature preservation. The proposed approach enables the scaling up of NIR datasets without additional data acquisition or annotation efforts, potentially accelerating advancements in NIR-based computer vision applications.
  arXiv  Detail & Related papers  (2024-09-25T07:51:47Z)
• Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer [10.982521876026281]
  We introduce a diffusion-based framework to address the RGB-D semantic segmentation problem. We demonstrate that utilizing a Deformable Attention Transformer as the encoder to extract features from depth images effectively captures the characteristics of invalid regions in depth measurements.
  arXiv  Detail & Related papers  (2024-09-23T15:23:01Z)
• Retinex-RAWMamba: Bridging Demosaicing and Denoising for Low-Light RAW Image Enhancement [71.13353154514418]
  Low-light image enhancement, particularly in cross-domain tasks such as mapping from the raw domain to the sRGB domain, remains a significant challenge. We present a novel Mamba scanning mechanism, called RAWMamba, to effectively handle raw images with different CFAs. We also present a Retinex Decomposition Module (RDM) grounded in Retinex prior, which decouples illumination from reflectance to facilitate more effective denoising and automatic non-linear exposure correction.
  arXiv  Detail & Related papers  (2024-09-11T06:12:03Z)
• Cross-Modality Perturbation Synergy Attack for Person Re-identification [66.48494594909123]
  The main challenge in cross-modality ReID lies in effectively dealing with visual differences between different modalities. Existing attack methods have primarily focused on the characteristics of the visible image modality. This study proposes a universal perturbation attack specifically designed for cross-modality ReID.
  arXiv  Detail & Related papers  (2024-01-18T15:56:23Z)
• Frequency Domain Modality-invariant Feature Learning for Visible-infrared Person Re-Identification [79.9402521412239]
  We propose a novel Frequency Domain modality-invariant feature learning framework (FDMNet) to reduce modality discrepancy from the frequency domain perspective. Our framework introduces two novel modules, namely the Instance-Adaptive Amplitude Filter (IAF) and the Phrase-Preserving Normalization (PPNorm)
  arXiv  Detail & Related papers  (2024-01-03T17:11:27Z)
• Breaking Modality Disparity: Harmonized Representation for Infrared and Visible Image Registration [66.33746403815283]
  We propose a scene-adaptive infrared and visible image registration. We employ homography to simulate the deformation between different planes. We propose the first ground truth available misaligned infrared and visible image dataset.
  arXiv  Detail & Related papers  (2023-04-12T06:49:56Z)
• Translation, Scale and Rotation: Cross-Modal Alignment Meets RGB-Infrared Vehicle Detection [10.460296317901662]
  We find detection in aerial RGB-IR images suffers from cross-modal weakly misalignment problems. We propose a Translation-Scale-Rotation Alignment (TSRA) module to address the problem. A two-stream feature alignment detector (TSFADet) based on the TSRA module is constructed for RGB-IR object detection in aerial images.
  arXiv  Detail & Related papers  (2022-09-28T03:06:18Z)
• Visible-Infrared Person Re-Identification Using Privileged Intermediate Information [10.816003787786766]
  Cross-modal person re-identification (ReID) is challenging due to the large domain shift in data distributions between RGB and IR modalities. This paper introduces a novel approach for a creating intermediate virtual domain that acts as bridges between the two main domains. We devised a new method to generate images between visible and infrared domains that provide additional information to train a deep ReID model.
  arXiv  Detail & Related papers  (2022-09-19T21:08:14Z)
• Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-Identification [84.32086702849338]
  We propose a novel modality-adaptive mixup and invariant decomposition (MID) approach for RGB-infrared person re-identification. MID designs a modality-adaptive mixup scheme to generate suitable mixed modality images between RGB and infrared images. Experiments on two challenging benchmarks demonstrate superior performance of MID over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-03-03T14:26:49Z)
• A Similarity Inference Metric for RGB-Infrared Cross-Modality Person Re-identification [66.49212581685127]
  Cross-modality person re-identification (re-ID) is a challenging task due to the large discrepancy between IR and RGB modalities. Existing methods address this challenge typically by aligning feature distributions or image styles across modalities. This paper presents a novel similarity inference metric (SIM) that exploits the intra-modality sample similarities to circumvent the cross-modality discrepancy.
  arXiv  Detail & Related papers  (2020-07-03T05:28:13Z)
• Cross-Spectrum Dual-Subspace Pairing for RGB-infrared Cross-Modality Person Re-Identification [15.475897856494583]
  Conventional person re-identification can only handle RGB color images, which will fail at dark conditions. RGB-infrared ReID (also known as Infrared-Visible ReID or Visible-Thermal ReID) is proposed. In this paper, a novel multi-spectrum image generation method is proposed and the generated samples are utilized to help the network to find discriminative information.
  arXiv  Detail & Related papers  (2020-02-29T09:01:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.