CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space

• URL: http://arxiv.org/abs/2609.00272v1
• Date: Mon, 31 Aug 2026 19:14:54 GMT
• Title: CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space
• Abstract summary: CrossFeat is a framework that enables an existing monomodal descriptor to operate across modalities.<n>Our method learns a crossing function in descriptor space that maps features from one modality to a representation compatible with another.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most advances in keypoint descriptions address monomodal settings, where image variations arise from viewpoint, illumination, or contrast changes. Multimodal scenarios involve images produced by fundamentally different sensing processes, such as multispectral imaging, RGB-depth, satellite imagery, or medical imaging, causing the same structures to appear differently. A common solution to cross-modal description is to train descriptors for each modality pair, which requires retraining whenever the modalities change, or to train large models, which incur a significant increase in runtime. Instead, we propose CrossFeat, a framework that enables an existing monomodal descriptor to operate across modalities. Our method learns a crossing function in descriptor space that maps features from one modality to a representation compatible with another. To preserve the structural information captured by the original descriptor, CrossFeat introduces a geometry-appearance disentanglement such that only appearance is altered while the geometric properties are preserved. Experiments across multiple domains and datasets demonstrate improved performance in multimodal matching.

Related papers

• Self-Supervised Spatial Correspondence Across Modalities [17.50529887238381]
  We present a method for finding cross-modal space-time correspondences.<n>Given two images, such as an RGB image and a depth map, our model identifies which pairs of pixels correspond to the same physical points in the scene.
  arXiv  Detail & Related papers  (2025-06-03T17:59:45Z)
• FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens [56.752362642658504]
  We present FuseLIP, an alternative architecture for multimodal embedding.<n>We propose a single transformer model which operates on an extended vocabulary of text and image tokens.<n>We show that FuseLIP outperforms other approaches in multimodal embedding tasks such as VQA and text-guided image transformation retrieval.
  arXiv  Detail & Related papers  (2025-06-03T17:27:12Z)
• CrossModalityDiffusion: Multi-Modal Novel View Synthesis with Unified Intermediate Representation [0.5242869847419834]
  CrossModalityDiffusion is a modular framework designed to generate images across different modalities without prior knowledge of scene geometry.<n>We show that jointly training different modules ensures consistent geometric understanding across all modalities within the framework.<n>We validate CrossModalityDiffusion's capabilities on the synthetic ShapeNet cars dataset.
  arXiv  Detail & Related papers  (2025-01-16T20:56:32Z)
• Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment [20.902935570581207]
  We introduce a Multimodal Alignment and Reconstruction Network (MARNet) to enhance the model's resistance to visual noise. MARNet includes a cross-modal diffusion reconstruction module for smoothly and stably blending information across different domains. Experiments conducted on two benchmark datasets, Vireo-Food172 and Ingredient-101, demonstrate that MARNet effectively improves the quality of image information extracted by the model.
  arXiv  Detail & Related papers  (2024-07-26T16:30:18Z)
• Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioning [71.14084801851381]
  Change captioning aims to succinctly describe the semantic change between a pair of similar images. Most existing methods directly capture the difference between them, which risk obtaining error-prone difference features. We propose a distractors-immune representation learning network that correlates the corresponding channels of two image representations.
  arXiv  Detail & Related papers  (2024-07-16T13:00:33Z)
• A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers [0.0]
  We propose a simple framework to achieve fair image segmentation of multiple modalities using a single conditional model. We show that our framework outperforms other cross-modality segmentation methods on the Multi-Modality Whole Heart Conditional Challenge.
  arXiv  Detail & Related papers  (2023-10-09T09:51:44Z)
• Unbiased Multi-Modality Guidance for Image Inpainting [27.286351511243502]
  We develop an end-to-end multi-modality guided transformer network for image inpainting. Within each transformer block, the proposed spatial-aware attention module can learn the multi-modal structural features efficiently. Our method enriches semantically consistent context in an image based on discriminative information from multiple modalities.
  arXiv  Detail & Related papers  (2022-08-25T03:13:43Z)
• End-to-end Multi-modal Video Temporal Grounding [105.36814858748285]
  We propose a multi-modal framework to extract complementary information from videos. We adopt RGB images for appearance, optical flow for motion, and depth maps for image structure. We conduct experiments on the Charades-STA and ActivityNet Captions datasets, and show that the proposed method performs favorably against state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-07-12T17:58:10Z)
• Exploring Modality-shared Appearance Features and Modality-invariant Relation Features for Cross-modality Person Re-Identification [72.95858515157603]
  Cross-modality person re-identification works rely on discriminative modality-shared features. Despite some initial success, such modality-shared appearance features cannot capture enough modality-invariant information. A novel cross-modality quadruplet loss is proposed to further reduce the cross-modality variations.
  arXiv  Detail & Related papers  (2021-04-23T11:14:07Z)
• Cross-Descriptor Visual Localization and Mapping [81.16435356103133]
  Visual localization and mapping is the key technology underlying the majority of Mixed Reality and robotics systems. We present three novel scenarios for localization and mapping which require the continuous update of feature representations. Our data-driven approach is agnostic to the feature descriptor type, has low computational requirements, and scales linearly with the number of description algorithms.
  arXiv  Detail & Related papers  (2020-12-02T18:19:51Z)
• SMILE: Semantically-guided Multi-attribute Image and Layout Editing [154.69452301122175]
  Attribute image manipulation has been a very active topic since the introduction of Generative Adversarial Networks (GANs) We present a multimodal representation that handles all attributes, be it guided by random noise or images, while only using the underlying domain information of the target domain. Our method is capable of adding, removing or changing either fine-grained or coarse attributes by using an image as a reference or by exploring the style distribution space.
  arXiv  Detail & Related papers  (2020-10-05T20:15:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.