Descriptive Image-Text Matching with Graded Contextual Similarity

• URL: http://arxiv.org/abs/2505.09997v2
• Date: Fri, 16 May 2025 05:08:35 GMT
• Title: Descriptive Image-Text Matching with Graded Contextual Similarity
• Authors: Jinhyun Jang, Jiyoung Lee, Kwanghoon Sohn,
• Abstract summary: We propose descriptive image-text matching, called DITM, to learn the graded contextual similarity between image and text.<n>We formulate the descriptiveness score of each sentence with cumulative term frequency-inverse document frequency (TF-IDF) to balance the pairwise similarity.<n>Our method leverages sentence descriptiveness to learn robust image-text matching in two key ways.
• Score: 41.10869519062159
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image-text matching aims to build correspondences between visual and textual data by learning their pairwise similarities. Most existing approaches have adopted sparse binary supervision, indicating whether a pair of images and sentences matches or not. However, such sparse supervision covers a limited subset of image-text relationships, neglecting their inherent many-to-many correspondences; an image can be described in numerous texts at different descriptive levels. Moreover, existing approaches overlook the implicit connections from general to specific descriptions, which form the underlying rationale for the many-to-many relationships between vision and language. In this work, we propose descriptive image-text matching, called DITM, to learn the graded contextual similarity between image and text by exploring the descriptive flexibility of language. We formulate the descriptiveness score of each sentence with cumulative term frequency-inverse document frequency (TF-IDF) to balance the pairwise similarity according to the keywords in the sentence. Our method leverages sentence descriptiveness to learn robust image-text matching in two key ways: (1) to refine the false negative labeling, dynamically relaxing the connectivity between positive and negative pairs, and (2) to build more precise matching, aligning a set of relevant sentences in a generic-to-specific order. By moving beyond rigid binary supervision, DITM enhances the discovery of both optimal matches and potential positive pairs. Extensive experiments on MS-COCO, Flickr30K, and CxC datasets demonstrate the effectiveness of our method in representing complex image-text relationships compared to state-of-the-art approaches. In addition, DITM enhances the hierarchical reasoning ability of the model, supported by the extensive analysis on HierarCaps benchmark.

Related papers

• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training [33.78990448307792]
  Image-text retrieval is a central problem for understanding the semantic relationship between vision and language. Previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework.
  arXiv  Detail & Related papers  (2023-06-15T00:19:13Z)
• Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality [50.48859793121308]
  Contrastively trained vision-language models have achieved remarkable progress in vision and language representation learning. Recent research has highlighted severe limitations in their ability to perform compositional reasoning over objects, attributes, and relations.
  arXiv  Detail & Related papers  (2023-05-23T08:28:38Z)
• Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning [0.65268245109828]
  Coherent entity-aware multi-image captioning aims to generate coherent captions for neighboring images in a news document. This paper proposes a coherent entity-aware multi-image captioning model by making use of coherence relationships.
  arXiv  Detail & Related papers  (2023-02-04T07:50:31Z)
• Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks [13.80433764370972]
  Text-to-image synthesis aims to generate a photo-realistic and semantic consistent image from a specific text description. We propose a novel Vision-Language Matching strategy for text-to-image synthesis, named VLMGAN*. The proposed dual multi-level vision-language matching strategy can be applied to other text-to-image synthesis methods.
  arXiv  Detail & Related papers  (2022-08-20T03:34:04Z)
• Two-stream Hierarchical Similarity Reasoning for Image-text Matching [66.43071159630006]
  A hierarchical similarity reasoning module is proposed to automatically extract context information. Previous approaches only consider learning single-stream similarity alignment. A two-stream architecture is developed to decompose image-text matching into image-to-text level and text-to-image level similarity computation.
  arXiv  Detail & Related papers  (2022-03-10T12:56:10Z)
• Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval [48.20798265640068]
  We introduce additional phrase-level supervision for the better identification of mismatched units in the text. We construct text scene graphs for the matched sentences and extract entities and triples as the phrase-level labels. For the training, we propose multi-scale matching losses from both global and local perspectives.
  arXiv  Detail & Related papers  (2021-09-12T14:21:15Z)
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching [69.34076386926984]
  Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations. We propose a Consensus-aware Visual-Semantic Embedding model to incorporate the consensus information.
  arXiv  Detail & Related papers  (2020-07-17T10:22:57Z)
• Improving Image Captioning with Better Use of Captions [65.39641077768488]
  We present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences.
  arXiv  Detail & Related papers  (2020-06-21T14:10:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.