UFM: Unified Feature Matching Pre-training with Multi-Modal Image Assistants

• URL: http://arxiv.org/abs/2503.21820v1
• Date: Wed, 26 Mar 2025 06:20:52 GMT
• Title: UFM: Unified Feature Matching Pre-training with Multi-Modal Image Assistants
• Authors: Yide Di, Yun Liao, Hao Zhou, Kaijun Zhu, Qing Duan, Junhui Liu, Mingyu Lu,
• Abstract summary: We introduce a Unified Feature Matching pre-trained model (UFM) designed to address feature matching challenges across a wide spectrum of modal images.<n>We present Multimodal Image Assistant (MIA) transformers, finely tunable structures adept at handling diverse feature matching problems.
• Score: 12.756326600787629
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image feature matching, a foundational task in computer vision, remains challenging for multimodal image applications, often necessitating intricate training on specific datasets. In this paper, we introduce a Unified Feature Matching pre-trained model (UFM) designed to address feature matching challenges across a wide spectrum of modal images. We present Multimodal Image Assistant (MIA) transformers, finely tunable structures adept at handling diverse feature matching problems. UFM exhibits versatility in addressing both feature matching tasks within the same modal and those across different modals. Additionally, we propose a data augmentation algorithm and a staged pre-training strategy to effectively tackle challenges arising from sparse data in specific modals and imbalanced modal datasets. Experimental results demonstrate that UFM excels in generalization and performance across various feature matching tasks. The code will be released at:https://github.com/LiaoYun0x0/UFM.

Related papers

• MIFNet: Learning Modality-Invariant Features for Generalizable Multimodal Image Matching [54.740256498985026]
  Keypoint detection and description methods often struggle with multimodal data.<n>We propose a modality-invariant feature learning network (MIFNet) to compute modality-invariant features for keypoint descriptions in multimodal image matching.
  arXiv  Detail & Related papers  (2025-01-20T06:56:30Z)
• MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training [62.843316348659165]
  Deep learning-based image matching algorithms have dramatically outperformed humans in rapidly and accurately finding large amounts of correspondences.<n>We propose a large-scale pre-training framework that utilizes synthetic cross-modal training signals to train models to recognize and match fundamental structures across images.<n>Our key finding is that the matching model trained with our framework achieves remarkable generalizability across more than eight unseen cross-modality registration tasks.
  arXiv  Detail & Related papers  (2025-01-13T18:37:36Z)
• MINIMA: Modality Invariant Image Matching [52.505282811925454]
  We present MINIMA, a unified image matching framework for multiple cross-modal cases.<n>We scale up the modalities from cheap but rich RGB-only matching data, by means of generative models.<n>With MD-syn, we can directly train any advanced matching pipeline on randomly selected modality pairs to obtain cross-modal ability.
  arXiv  Detail & Related papers  (2024-12-27T02:39:50Z)
• Task-Customized Mixture of Adapters for General Image Fusion [51.8742437521891]
  General image fusion aims at integrating important information from multi-source images. We propose a novel task-customized mixture of adapters (TC-MoA) for general image fusion, adaptively prompting various fusion tasks in a unified model.
  arXiv  Detail & Related papers  (2024-03-19T07:02:08Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training [79.78201886156513]
  We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently. Our approach achieves comparable results to single-task state-of-the-art models and demonstrates strong generalization on downstream tasks.
  arXiv  Detail & Related papers  (2023-06-29T17:59:57Z)
• MultiMAE: Multi-modal Multi-task Masked Autoencoders [2.6763498831034043]
  We propose a pre-training strategy called Multi-modal Multi-task Masked Autoencoders (MultiMAE) We show this pre-training strategy leads to a flexible, simple, and efficient framework with improved transfer results to downstream tasks.
  arXiv  Detail & Related papers  (2022-04-04T17:50:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.