Improving Text-based Person Search via Part-level Cross-modal Correspondence

• URL: http://arxiv.org/abs/2501.00318v1
• Date: Tue, 31 Dec 2024 07:29:50 GMT
• Title: Improving Text-based Person Search via Part-level Cross-modal Correspondence
• Authors: Jicheol Park, Boseung Jeong, Dongwon Kim, Suha Kwak,
• Abstract summary: We introduce an efficient encoder-decoder model that extracts coarse-to-fine embedding vectors. There is another challenge of learning to capture fine-grained information with only person IDs as supervision. We propose a novel ranking loss, dubbed commonality-based margin ranking loss, which quantifies the degree of commonality of each body part.
• Score: 29.301950609839796
• License:
• Abstract: Text-based person search is the task of finding person images that are the most relevant to the natural language text description given as query. The main challenge of this task is a large gap between the target images and text queries, which makes it difficult to establish correspondence and distinguish subtle differences across people. To address this challenge, we introduce an efficient encoder-decoder model that extracts coarse-to-fine embedding vectors which are semantically aligned across the two modalities without supervision for the alignment. There is another challenge of learning to capture fine-grained information with only person IDs as supervision, where similar body parts of different individuals are considered different due to the lack of part-level supervision. To tackle this, we propose a novel ranking loss, dubbed commonality-based margin ranking loss, which quantifies the degree of commonality of each body part and reflects it during the learning of fine-grained body part details. As a consequence, it enables our method to achieve the best records on three public benchmarks.

Related papers

• Efficient Vision Language Model Fine-tuning for Text-based Person Anomaly Search [20.695290280579858]
  This paper presents the HFUT-LMC team's solution to the WWW 2025 challenge on Text-based Person Anomaly Search (TPAS) The primary objective of this challenge is to accurately identify pedestrians exhibiting either normal or abnormal behavior within a large library of pedestrian images. We introduce the Similarity Coverage Analysis (SCA) strategy to address the recognition difficulty caused by similar text descriptions.
  arXiv  Detail & Related papers  (2025-02-05T14:45:09Z)
• Knowing Where to Focus: Attention-Guided Alignment for Text-based Person Search [64.15205542003056]
  We introduce Attention-Guided Alignment (AGA) framework featuring two innovative components: Attention-Guided Mask (AGM) Modeling and Text Enrichment Module (TEM) AGA achieves new state-of-the-art results with Rank-1 accuracy reaching 78.36%, 67.31%, and 67.4% on CUHK-PEDES, ICFG-PEDES, and RSTP, respectively.
  arXiv  Detail & Related papers  (2024-12-19T17:51:49Z)
• PLOT: Text-based Person Search with Part Slot Attention for Corresponding Part Discovery [29.301950609839796]
  We propose a novel framework that leverages a part discovery module based on slot attention to autonomously identify and align distinctive parts across modalities. Our method is evaluated on three public benchmarks, significantly outperforming existing methods.
  arXiv  Detail & Related papers  (2024-09-20T13:05:55Z)
• Shatter and Gather: Learning Referring Image Segmentation with Text Supervision [52.46081425504072]
  We present a new model that discovers semantic entities in input image and then combines such entities relevant to text query to predict the mask of the referent. Our method was evaluated on four public benchmarks for referring image segmentation, where it clearly outperformed the existing method for the same task and recent open-vocabulary segmentation models on all the benchmarks.
  arXiv  Detail & Related papers  (2023-08-29T15:39:15Z)
• Text-guided Image Restoration and Semantic Enhancement for Text-to-Image Person Retrieval [12.057465578064345]
  The goal of Text-to-Image Person Retrieval (TIPR) is to retrieve specific person images according to the given textual descriptions. We propose a novel TIPR framework to build fine-grained interactions and alignment between person images and the corresponding texts.
  arXiv  Detail & Related papers  (2023-07-18T08:23:46Z)
• TextFormer: A Query-based End-to-End Text Spotter with Mixed Supervision [61.186488081379]
  We propose TextFormer, a query-based end-to-end text spotter with Transformer architecture. TextFormer builds upon an image encoder and a text decoder to learn a joint semantic understanding for multi-task modeling. It allows for mutual training and optimization of classification, segmentation, and recognition branches, resulting in deeper feature sharing.
  arXiv  Detail & Related papers  (2023-06-06T03:37:41Z)
• Three ways to improve feature alignment for open vocabulary detection [88.65076922242184]
  Key problem in zero-shot open vocabulary detection is how to align visual and text features, so that the detector performs well on unseen classes. Previous approaches train the feature pyramid and detection head from scratch, which breaks the vision-text feature alignment established during pretraining. We propose three methods to alleviate these issues. Firstly, a simple scheme is used to augment the text embeddings which prevents overfitting to a small number of classes seen during training. Secondly, the feature pyramid network and the detection head are modified to include trainable shortcuts. Finally, a self-training approach is used to leverage a larger corpus of
  arXiv  Detail & Related papers  (2023-03-23T17:59:53Z)
• Image-Specific Information Suppression and Implicit Local Alignment for Text-based Person Search [61.24539128142504]
  Text-based person search (TBPS) is a challenging task that aims to search pedestrian images with the same identity from an image gallery given a query text. Most existing methods rely on explicitly generated local parts to model fine-grained correspondence between modalities. We propose an efficient joint Multi-level Alignment Network (MANet) for TBPS, which can learn aligned image/text feature representations between modalities at multiple levels.
  arXiv  Detail & Related papers  (2022-08-30T16:14:18Z)
• Learning Semantic-Aligned Feature Representation for Text-based Person Search [8.56017285139081]
  We propose a semantic-aligned embedding method for text-based person search. The feature alignment across modalities is achieved by automatically learning the semantic-aligned visual features and textual features. Experimental results on the CUHK-PEDES and Flickr30K datasets show that our method achieves state-of-the-art performances.
  arXiv  Detail & Related papers  (2021-12-13T14:54:38Z)
• Text-based Person Search in Full Images via Semantic-Driven Proposal Generation [42.25611020956918]
  We propose a new end-to-end learning framework which jointly optimize the pedestrian detection, identification and visual-semantic feature embedding tasks. To take full advantage of the query text, the semantic features are leveraged to instruct the Region Proposal Network to pay more attention to the text-described proposals.
  arXiv  Detail & Related papers  (2021-09-27T11:42:40Z)
• Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification [78.45528514468836]
  Text-to-image person re-identification (ReID) aims to search for images containing a person of interest using textual descriptions. We propose a Semantically Self-Aligned Network (SSAN) to handle the above problems. To expedite future research in text-to-image ReID, we build a new database named ICFG-PEDES.
  arXiv  Detail & Related papers  (2021-07-27T08:26:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.