Embedding and Enriching Explicit Semantics for Visible-Infrared Person Re-Identification

• URL: http://arxiv.org/abs/2412.08406v1
• Date: Wed, 11 Dec 2024 14:27:30 GMT
• Title: Embedding and Enriching Explicit Semantics for Visible-Infrared Person Re-Identification
• Authors: Neng Dong, Shuanglin Yan, Liyan Zhang, Jinhui Tang,
• Abstract summary: Visible-infrared person re-identification (VIReID) retrieves pedestrian images with the same identity across different modalities. Existing methods learn visual content solely from images, lacking the capability to sense high-level semantics. We propose an Embedding and Enriching Explicit Semantics framework to learn semantically rich cross-modality pedestrian representations.
• Score: 31.011118085494942
• License:
• Abstract: Visible-infrared person re-identification (VIReID) retrieves pedestrian images with the same identity across different modalities. Existing methods learn visual content solely from images, lacking the capability to sense high-level semantics. In this paper, we propose an Embedding and Enriching Explicit Semantics (EEES) framework to learn semantically rich cross-modality pedestrian representations. Our method offers several contributions. First, with the collaboration of multiple large language-vision models, we develop Explicit Semantics Embedding (ESE), which automatically supplements language descriptions for pedestrians and aligns image-text pairs into a common space, thereby learning visual content associated with explicit semantics. Second, recognizing the complementarity of multi-view information, we present Cross-View Semantics Compensation (CVSC), which constructs multi-view image-text pair representations, establishes their many-to-many matching, and propagates knowledge to single-view representations, thus compensating visual content with its missing cross-view semantics. Third, to eliminate noisy semantics such as conflicting color attributes in different modalities, we design Cross-Modality Semantics Purification (CMSP), which constrains the distance between inter-modality image-text pair representations to be close to that between intra-modality image-text pair representations, further enhancing the modality-invariance of visual content. Finally, experimental results demonstrate the effectiveness and superiority of the proposed EEES.

Related papers

• MVAM: Multi-View Attention Method for Fine-grained Image-Text Matching [65.87255122130188]
  We propose a Multi-view Attention Method (MVAM) for image-text matching. We also incorporate an objective to explicitly encourage attention heads to focus on distinct aspects of the input data. Our method allows models to encode images and text from different perspectives and focus on more critical details, leading to better matching performance.
  arXiv  Detail & Related papers  (2024-02-27T06:11:54Z)
• CLIP-Driven Semantic Discovery Network for Visible-Infrared Person Re-Identification [39.262536758248245]
  Cross-modality identity matching poses significant challenges in VIReID. We propose a CLIP-Driven Semantic Discovery Network (CSDN) that consists of Modality-specific Prompt Learner, Semantic Information Integration, and High-level Semantic Embedding.
  arXiv  Detail & Related papers  (2024-01-11T10:20:13Z)
• Collaborative Group: Composed Image Retrieval via Consensus Learning from Noisy Annotations [67.92679668612858]
  We propose the Consensus Network (Css-Net), inspired by the psychological concept that groups outperform individuals. Css-Net comprises two core components: (1) a consensus module with four diverse compositors, each generating distinct image-text embeddings; and (2) a Kullback-Leibler divergence loss that encourages learning of inter-compositor interactions. On benchmark datasets, particularly FashionIQ, Css-Net demonstrates marked improvements. Notably, it achieves significant recall gains, with a 2.77% increase in R@10 and 6.67% boost in R@50, underscoring its
  arXiv  Detail & Related papers  (2023-06-03T11:50:44Z)
• Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens [76.40196364163663]
  We propose a learning-based vision-language pre-training approach, such as CLIP. We show that our method can learn more comprehensive representations and capture meaningful cross-modal correspondence.
  arXiv  Detail & Related papers  (2023-03-27T00:58:39Z)
• CISum: Learning Cross-modality Interaction to Enhance Multimodal Semantic Coverage for Multimodal Summarization [2.461695698601437]
  This paper proposes a multi-task cross-modality learning framework (CISum) to improve multimodal semantic coverage. To obtain the visual semantics, we translate images into visual descriptions based on the correlation with text content. Then, the visual description and text content are fused to generate the textual summary to capture the semantics of the multimodal content.
  arXiv  Detail & Related papers  (2023-02-20T11:57:23Z)
• Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Media [11.235498285650142]
  Named Entity Recognition (NER) on social media refers to discovering and classifying entities from unstructured free-form content. This work introduces the multi-granularity cross-modality representation learning. Experiments show that our proposed approach can achieve the SOTA or approximate SOTA performance on two benchmark datasets of tweets.
  arXiv  Detail & Related papers  (2022-10-19T15:14:55Z)
• Fine-Grained Semantically Aligned Vision-Language Pre-Training [151.7372197904064]
  Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts. We introduce LO, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions.
  arXiv  Detail & Related papers  (2022-08-04T07:51:48Z)
• Matching Visual Features to Hierarchical Semantic Topics for Image Paragraph Captioning [50.08729005865331]
  This paper develops a plug-and-play hierarchical-topic-guided image paragraph generation framework. To capture the correlations between the image and text at multiple levels of abstraction, we design a variational inference network. To guide the paragraph generation, the learned hierarchical topics and visual features are integrated into the language model.
  arXiv  Detail & Related papers  (2021-05-10T06:55:39Z)
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching [69.34076386926984]
  Image-text matching plays a central role in bridging vision and language. Most existing approaches only rely on the image-text instance pair to learn their representations. We propose a Consensus-aware Visual-Semantic Embedding model to incorporate the consensus information.
  arXiv  Detail & Related papers  (2020-07-17T10:22:57Z)
• MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding [6.4901484665257545]
  We propose a novel multi-head self-attention network to capture various components of visual and textual data by attending to important parts in data. Our approach achieves the new state-of-the-art results in image-text retrieval tasks on MS-COCO and Flicker30K datasets.
  arXiv  Detail & Related papers  (2020-01-11T05:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.