Breaking Modality Disparity: Harmonized Representation for Infrared and
Visible Image Registration
- URL: http://arxiv.org/abs/2304.05646v2
- Date: Mon, 27 Nov 2023 08:12:14 GMT
- Title: Breaking Modality Disparity: Harmonized Representation for Infrared and
Visible Image Registration
- Authors: Zhiying Jiang, Zengxi Zhang, Jinyuan Liu, Xin Fan, Risheng Liu
- Abstract summary: We propose a scene-adaptive infrared and visible image registration.
We employ homography to simulate the deformation between different planes.
We propose the first ground truth available misaligned infrared and visible image dataset.
- Score: 66.33746403815283
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Since the differences in viewing range, resolution and relative position, the
multi-modality sensing module composed of infrared and visible cameras needs to
be registered so as to have more accurate scene perception. In practice, manual
calibration-based registration is the most widely used process, and it is
regularly calibrated to maintain accuracy, which is time-consuming and
labor-intensive. To cope with these problems, we propose a scene-adaptive
infrared and visible image registration. Specifically, in regard of the
discrepancy between multi-modality images, an invertible translation process is
developed to establish a modality-invariant domain, which comprehensively
embraces the feature intensity and distribution of both infrared and visible
modalities. We employ homography to simulate the deformation between different
planes and develop a hierarchical framework to rectify the deformation inferred
from the proposed latent representation in a coarse-to-fine manner. For that,
the advanced perception ability coupled with the residual estimation conducive
to the regression of sparse offsets, and the alternate correlation search
facilitates a more accurate correspondence matching. Moreover, we propose the
first ground truth available misaligned infrared and visible image dataset,
involving three synthetic sets and one real-world set. Extensive experiments
validate the effectiveness of the proposed method against the
state-of-the-arts, advancing the subsequent applications.
Related papers
- Improving Misaligned Multi-modality Image Fusion with One-stage
Progressive Dense Registration [67.23451452670282]
Misalignments between multi-modality images pose challenges in image fusion.
We propose a Cross-modality Multi-scale Progressive Dense Registration scheme.
This scheme accomplishes the coarse-to-fine registration exclusively using a one-stage optimization.
arXiv Detail & Related papers (2023-08-22T03:46:24Z) - View Consistent Purification for Accurate Cross-View Localization [59.48131378244399]
This paper proposes a fine-grained self-localization method for outdoor robotics.
The proposed method addresses limitations in existing cross-view localization methods.
It is the first sparse visual-only method that enhances perception in dynamic environments.
arXiv Detail & Related papers (2023-08-16T02:51:52Z) - Unsupervised Misaligned Infrared and Visible Image Fusion via
Cross-Modality Image Generation and Registration [59.02821429555375]
We present a robust cross-modality generation-registration paradigm for unsupervised misaligned infrared and visible image fusion.
To better fuse the registered infrared images and visible images, we present a feature Interaction Fusion Module (IFM)
arXiv Detail & Related papers (2022-05-24T07:51:57Z) - ReDFeat: Recoupling Detection and Description for Multimodal Feature
Learning [51.07496081296863]
We recouple independent constraints of detection and description of multimodal feature learning with a mutual weighting strategy.
We propose a detector that possesses a large receptive field and is equipped with learnable non-maximum suppression layers.
We build a benchmark that contains cross visible, infrared, near-infrared and synthetic aperture radar image pairs for evaluating the performance of features in feature matching and image registration tasks.
arXiv Detail & Related papers (2022-05-16T04:24:22Z) - Towards Homogeneous Modality Learning and Multi-Granularity Information
Exploration for Visible-Infrared Person Re-Identification [16.22986967958162]
Visible-infrared person re-identification (VI-ReID) is a challenging and essential task, which aims to retrieve a set of person images over visible and infrared camera views.
Previous methods attempt to apply generative adversarial network (GAN) to generate the modality-consisitent data.
In this work, we address cross-modality matching problem with Aligned Grayscale Modality (AGM), an unified dark-line spectrum that reformulates visible-infrared dual-mode learning as a gray-gray single-mode learning problem.
arXiv Detail & Related papers (2022-04-11T03:03:19Z) - Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared
Person Re-Identification [84.32086702849338]
We propose a novel modality-adaptive mixup and invariant decomposition (MID) approach for RGB-infrared person re-identification.
MID designs a modality-adaptive mixup scheme to generate suitable mixed modality images between RGB and infrared images.
Experiments on two challenging benchmarks demonstrate superior performance of MID over state-of-the-art methods.
arXiv Detail & Related papers (2022-03-03T14:26:49Z) - DiffuseMorph: Unsupervised Deformable Image Registration Along
Continuous Trajectory Using Diffusion Models [31.826844124173984]
We present a novel approach of diffusion model-based probabilistic image registration, called DiffuseMorph.
Our model learns the score function of the deformation between moving and fixed images.
Our method can provide flexible and accurate deformation with a capability of topology preservation.
arXiv Detail & Related papers (2021-12-09T08:41:23Z) - Unsupervised Multimodal Image Registration with Adaptative Gradient
Guidance [23.461130560414805]
Unsupervised learning-based methods have demonstrated promising performance over accuracy and efficiency in deformable image registration.
The estimated deformation fields of the existing methods fully rely on the to-be-registered image pair.
We propose a novel multimodal registration framework, which leverages the deformation fields estimated from both.
arXiv Detail & Related papers (2020-11-12T05:47:20Z) - CoMIR: Contrastive Multimodal Image Representation for Registration [4.543268895439618]
We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations)
CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures.
arXiv Detail & Related papers (2020-06-11T10:51:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.