OSDM-MReg: Multimodal Image Registration based One Step Diffusion Model

• URL: http://arxiv.org/abs/2504.06027v2
• Date: Mon, 15 Sep 2025 04:32:39 GMT
• Title: OSDM-MReg: Multimodal Image Registration based One Step Diffusion Model
• Authors: Xiaochen Wei, Weiwei Guo, Wenxian Yu, Feiming Wei, Dongying Li,
• Abstract summary: Multimodal remote sensing image registration aligns images from different sensors for data fusion and analysis.<n>We propose OSDM-MReg, a novel multimodal image registration framework that bridges the modality gap through image-to-image translation.
• Score: 16.850096473419505
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal remote sensing image registration aligns images from different sensors for data fusion and analysis. However, existing methods often struggle to extract modality-invariant features when faced with large nonlinear radiometric differences, such as those between SAR and optical images. To address these challenges, we propose OSDM-MReg, a novel multimodal image registration framework that bridges the modality gap through image-to-image translation. Specifically, we introduce a one-step unaligned target-guided conditional diffusion model (UTGOS-CDM) to translate source and target images into a unified representation domain. Unlike traditional conditional DDPM that require hundreds of iterative steps for inference, our model incorporates a novel inverse translation objective during training to enable direct prediction of the translated image in a single step at test time, significantly accelerating the registration process. After translation, we design a multimodal multiscale registration network (MM-Reg) that extracts and fuses both unimodal and translated multimodal images using the proposed multimodal fusion strategy, enhancing the robustness and precision of alignment across scales and modalities. Extensive experiments on the OSdataset demonstrate that OSDM-MReg achieves superior registration accuracy compared to state-of-the-art methods.

Related papers

• Multimodal Feature Fusion Network with Text Difference Enhancement for Remote Sensing Change Detection [36.96267014127019]
  MMChange is a multimodal RSCD method that combines image and text modalities to enhance accuracy and robustness.<n>To overcome the semantic limitations of image features, we employ a vision language model (VLM) to generate semantic descriptions of bitemporal images.<n>A Textual Difference Enhancement (TDE) module captures fine grained semantic shifts, guiding the model toward meaningful changes.
  arXiv  Detail & Related papers  (2025-09-04T07:39:18Z)
• FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens [56.752362642658504]
  We present FuseLIP, an alternative architecture for multimodal embedding.<n>We propose a single transformer model which operates on an extended vocabulary of text and image tokens.<n>We show that FuseLIP outperforms other approaches in multimodal embedding tasks such as VQA and text-guided image transformation retrieval.
  arXiv  Detail & Related papers  (2025-06-03T17:27:12Z)
• Bi-directional Self-Registration for Misaligned Infrared-Visible Image Fusion [53.60977801655896]
  Acquiring accurately aligned multi-modal image pairs is fundamental for achieving high-quality multi-modal image fusion.<n>We propose a novel self-supervised textbfBi-directional textbfSelf-textbfRegistration framework (textbfB-SR)
  arXiv  Detail & Related papers  (2025-05-11T09:36:25Z)
• Unified Multimodal Discrete Diffusion [78.48930545306654]
  Multimodal generative models that can understand and generate across multiple modalities are dominated by autoregressive (AR) approaches.<n>We explore discrete diffusion models as a unified generative formulation in the joint text and image domain.<n>We present the first Unified Multimodal Discrete Diffusion (UniDisc) model which is capable of jointly understanding and generating text and images.
  arXiv  Detail & Related papers  (2025-03-26T17:59:51Z)
• MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching [54.740256498985026]
  Keypoint detection and description methods often struggle with multimodal data.<n>We propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching.
  arXiv  Detail & Related papers  (2025-01-20T06:56:30Z)
• EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM [38.8308841469793]
  This paper introduces EasyRef, a novel plug-and-play adaptation method that enables diffusion models to be conditioned on multiple reference images and the text prompt.<n>We leverage the multi-image comprehension and instruction-following capabilities of the multimodal large language model (MLLM) to exploit consistent visual elements within multiple images.<n> Experimental results demonstrate EasyRef surpasses both tuning-free methods like IP-Adapter and tuning-based methods like LoRA, achieving superior aesthetic quality and robust zero-shot generalization across diverse domains.
  arXiv  Detail & Related papers  (2024-12-12T18:59:48Z)
• Unsupervised Modality Adaptation with Text-to-Image Diffusion Models for Semantic Segmentation [54.96563068182733]
  We propose Modality Adaptation with text-to-image Diffusion Models (MADM) for semantic segmentation task. MADM utilizes text-to-image diffusion models pre-trained on extensive image-text pairs to enhance the model's cross-modality capabilities. We show that MADM achieves state-of-the-art adaptation performance across various modality tasks, including images to depth, infrared, and event modalities.
  arXiv  Detail & Related papers  (2024-10-29T03:49:40Z)
• MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling [64.09238330331195]
  We propose a novel Multi-Modal Auto-Regressive (MMAR) probabilistic modeling framework. Unlike discretization line of method, MMAR takes in continuous-valued image tokens to avoid information loss. We show that MMAR demonstrates much more superior performance than other joint multi-modal models.
  arXiv  Detail & Related papers  (2024-10-14T17:57:18Z)
• I2I-Galip: Unsupervised Medical Image Translation Using Generative Adversarial CLIP [30.506544165999564]
  Unpaired image-to-image translation is a challenging task due to the absence of paired examples. We propose a new image-to-image translation framework named Image-to-Image-Generative-Adversarial-CLIP (I2I-Galip)
  arXiv  Detail & Related papers  (2024-09-19T01:44:50Z)
• Large Language Models for Multimodal Deformable Image Registration [50.91473745610945]
  We propose a novel coarse-to-fine MDIR framework,LLM-Morph, for aligning the deep features from different modal medical images. Specifically, we first utilize a CNN encoder to extract deep visual features from cross-modal image pairs, then we use the first adapter to adjust these tokens, and use LoRA in pre-trained LLMs to fine-tune their weights. Third, for the alignment of tokens, we utilize other four adapters to transform the LLM-encoded tokens into multi-scale visual features, generating multi-scale deformation fields and facilitating the coarse-to-fine MDIR task
  arXiv  Detail & Related papers  (2024-08-20T09:58:30Z)
• Cross-Domain Separable Translation Network for Multimodal Image Change Detection [11.25422609271201]
  multimodal change detection (MCD) is particularly critical in the remote sensing community. This paper focuses on addressing the challenges of MCD, especially the difficulty in comparing images from different sensors. A novel unsupervised cross-domain separable translation network (CSTN) is proposed to overcome these limitations.
  arXiv  Detail & Related papers  (2024-07-23T03:56:02Z)
• Bridging the Gap between Synthetic and Authentic Images for Multimodal Machine Translation [51.37092275604371]
  Multimodal machine translation (MMT) simultaneously takes the source sentence and a relevant image as input for translation. Recent studies suggest utilizing powerful text-to-image generation models to provide image inputs. However, synthetic images generated by these models often follow different distributions compared to authentic images.
  arXiv  Detail & Related papers  (2023-10-20T09:06:30Z)
• MAD: Modality Agnostic Distance Measure for Image Registration [14.558286801723293]
  Multi-modal image registration is a crucial pre-processing step in many medical applications. We present Modality Agnostic Distance (MAD), a measure that uses random convolutions to learn the inherent geometry of the images. We demonstrate that not only can MAD affinely register multi-modal images successfully, but it has also a larger capture range than traditional measures.
  arXiv  Detail & Related papers  (2023-09-06T09:59:58Z)
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis [52.41439725865149]
  Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. We propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis.
  arXiv  Detail & Related papers  (2022-12-02T11:40:40Z)
• Semantic Image Synthesis via Diffusion Models [174.24523061460704]
  Denoising Diffusion Probabilistic Models (DDPMs) have achieved remarkable success in various image generation tasks. Recent work on semantic image synthesis mainly follows the de facto GAN-based approaches. We propose a novel framework based on DDPM for semantic image synthesis.
  arXiv  Detail & Related papers  (2022-06-30T18:31:51Z)
• BBDM: Image-to-image Translation with Brownian Bridge Diffusion Models [50.39417112077254]
  A novel image-to-image translation method based on the Brownian Bridge Diffusion Model (BBDM) is proposed. To the best of our knowledge, it is the first work that proposes Brownian Bridge diffusion process for image-to-image translation.
  arXiv  Detail & Related papers  (2022-05-16T13:47:02Z)
• Unsupervised Multi-Modal Medical Image Registration via Discriminator-Free Image-to-Image Translation [4.43142018105102]
  We propose a novel translation-based unsupervised deformable image registration approach to convert the multi-modal registration problem to a mono-modal one. Our approach incorporates a discriminator-free translation network to facilitate the training of the registration network and a patchwise contrastive loss to encourage the translation network to preserve object shapes.
  arXiv  Detail & Related papers  (2022-04-28T17:18:21Z)
• CoMIR: Contrastive Multimodal Image Representation for Registration [4.543268895439618]
  We propose contrastive coding to learn shared, dense image representations, referred to as CoMIRs (Contrastive Multimodal Image Representations) CoMIRs enable the registration of multimodal images where existing registration methods often fail due to a lack of sufficiently similar image structures.
  arXiv  Detail & Related papers  (2020-06-11T10:51:33Z)
• Unsupervised Multi-Modal Image Registration via Geometry Preserving Image-to-Image Translation [43.060971647266236]
  We train an image-to-image translation network on the two input modalities. This learned translation allows training the registration network using simple and reliable mono-modality metrics. Compared to state-of-the-art multi-modal methods our presented method is unsupervised, requiring no pairs of aligned modalities for training, and can be adapted to any pair of modalities.
  arXiv  Detail & Related papers  (2020-03-18T07:21:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.