Learning Image-Text Matching with Optimal Partial Transport

• URL: http://arxiv.org/abs/2603.14349v1
• Date: Sun, 15 Mar 2026 12:31:19 GMT
• Title: Learning Image-Text Matching with Optimal Partial Transport
• Abstract summary: Cross-modal matching, a fundamental task in bridging vision and language, has recently garnered substantial research interest.<n>We propose the crOss-Modal sInkhorn maTching network as an effective solution to effectively improve performance while maintaining efficiency.<n>We conduct extensive evaluations of OMIT on two benchmark image-text retrieval datasets, namely Flickr30K and MS-COCO.
• Score: 49.656836538448054
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-modal matching, a fundamental task in bridging vision and language, has recently garnered substantial research interest. Despite the development of numerous methods aimed at quantifying the semantic relatedness between image-text pairs, these methods often fall short of achieving both outstanding performance and high efficiency. In this paper, we propose the crOss-Modal sInkhorn maTching (OMIT) network as an effective solution to effectively improving performance while maintaining efficiency. Rooted in the theoretical foundations of Optimal Transport, OMIT harnesses the capabilities of Cross-modal Mover's Distance to precisely compute the similarity between fine-grained visual and textual fragments, utilizing Sinkhorn iterations for efficient approximation. To further alleviate the issue of redundant alignments, we seamlessly integrate partial matching into OMIT, leveraging local-to-global similarities to eliminate the interference of irrelevant fragments. We conduct extensive evaluations of OMIT on two benchmark image-text retrieval datasets, namely Flickr30K and MS-COCO. The superior performance achieved by OMIT on both datasets unequivocally demonstrates its effectiveness in cross-modal matching. Furthermore, through comprehensive visualization analysis, we elucidate OMIT's inherent tendency towards focal matching, thereby shedding light on its efficacy. Our code is publicly available at https://github.com/ppanzx/OMIT.

Related papers

• Fast-then-Fine: A Two-Stage Framework with Multi-Granular Representation for Cross-Modal Retrieval in Remote Sensing [13.181195595944603]
  Remote sensing (RS) image-text retrieval plays a critical role in understanding massive RS imagery.<n>Existing methods either rely on complex cross-modal interactions that lead to low retrieval efficiency, or depend on large-scale vision-language model pre-training.<n>We propose a fast-then-fine (FTF) two-stage retrieval framework that decomposes retrieval into a text-agnostic recall stage and a text-guided rerank stage for fine-grained alignment.
  arXiv  Detail & Related papers  (2026-04-22T10:50:38Z)
• Optimal Transport Adapter Tuning for Bridging Modality Gaps in Few-Shot Remote Sensing Scene Classification [80.83325513157637]
  Few-Shot Remote Sensing Scene Classification (FS-RSSC) presents the challenge of classifying remote sensing images with limited labeled samples.<n>We propose a novel Optimal Transport Adapter Tuning (OTAT) framework aimed at constructing an ideal Platonic representational space.
  arXiv  Detail & Related papers  (2025-03-19T07:04:24Z)
• VERITE: A Robust Benchmark for Multimodal Misinformation Detection Accounting for Unimodal Bias [17.107961913114778]
  multimodal misinformation is a growing problem on social media platforms. In this study, we investigate and identify the presence of unimodal bias in widely-used MMD benchmarks. We introduce a new method -- termed Crossmodal HArd Synthetic MisAlignment (CHASMA) -- for generating realistic synthetic training data.
  arXiv  Detail & Related papers  (2023-04-27T12:28:29Z)
• SCMM: Calibrating Cross-modal Representations for Text-Based Person Search [45.24784242117999]
  Text-Based Person Search (TBPS) faces critical challenges in cross-modal information fusion.<n>We propose SCMM (Sew and Masked Modeling), a novel framework addressing these fusion challenges through two complementary mechanisms.
  arXiv  Detail & Related papers  (2023-04-05T07:50:16Z)
• ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval [51.588385824875886]
  Cross-modal retrieval consists in finding images related to a given query text or vice-versa. Many recent methods proposed effective solutions to the image-text matching problem, mostly using recent large vision-language (VL) Transformer networks. This paper proposes an ALign And DIstill Network (ALADIN) to fill in the gap between effectiveness and efficiency.
  arXiv  Detail & Related papers  (2022-07-29T16:01:48Z)
• COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval [59.15034487974549]
  We propose a novel COllaborative Two-Stream vision-language pretraining model termed COTS for image-text retrieval. Our COTS achieves the highest performance among all two-stream methods and comparable performance with 10,800X faster in inference. Importantly, our COTS is also applicable to text-to-video retrieval, yielding new state-ofthe-art on the widely-used MSR-VTT dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:34:47Z)
• Learning Relation Alignment for Calibrated Cross-modal Retrieval [52.760541762871505]
  We propose a novel metric, Intra-modal Self-attention Distance (ISD), to quantify the relation consistency by measuring the semantic distance between linguistic and visual relations. We present Inter-modal Alignment on Intra-modal Self-attentions (IAIS), a regularized training method to optimize the ISD and calibrate intra-modal self-attentions mutually via inter-modal alignment.
  arXiv  Detail & Related papers  (2021-05-28T14:25:49Z)
• Weakly supervised cross-domain alignment with optimal transport [102.8572398001639]
  Cross-domain alignment between image objects and text sequences is key to many visual-language tasks. This paper investigates a novel approach for the identification and optimization of fine-grained semantic similarities between image and text entities.
  arXiv  Detail & Related papers  (2020-08-14T22:48:36Z)
• Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoders [14.634046503477979]
  We present a novel approach called Transformer Reasoning and Alignment Network (TERAN) TERAN enforces a fine-grained match between the underlying components of images and sentences. On the MS-COCO 1K test set, we obtain an improvement of 5.7% and 3.5% respectively on the image and the sentence retrieval tasks.
  arXiv  Detail & Related papers  (2020-08-12T11:02:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.