Calibrating Cross-modal Features for Text-Based Person Searching
- URL: http://arxiv.org/abs/2304.02278v2
- Date: Thu, 1 Jun 2023 01:49:26 GMT
- Title: Calibrating Cross-modal Features for Text-Based Person Searching
- Authors: Donglai Wei, Sipeng Zhang, Tong Yang, Yang Liu, Jing Liu
- Abstract summary: We present a simple yet effective method that calibrates cross-modal features from two perspectives.
Our method consists of two novel losses to provide fine-grained cross-modal features.
It achieves top results on three popular benchmarks with 73.81%, 74.25% and 57.35% Rank1 accuracy.
- Score: 18.3145271655619
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Text-Based Person Searching (TBPS) aims to identify the images of pedestrian
targets from a large-scale gallery with given textual caption. For cross-modal
TBPS task, it is critical to obtain well-distributed representation in the
common embedding space to reduce the inter-modal gap. Furthermore, it is also
essential to learn detailed image-text correspondence efficiently to
discriminate similar targets and enable fine-grained target search. To address
these challenges, we present a simple yet effective method that calibrates
cross-modal features from these two perspectives. Our method consists of two
novel losses to provide fine-grained cross-modal features. The Sew calibration
loss takes the quality of textual captions as guidance and aligns features
between image and text modalities. On the other hand, the Masking Caption
Modeling (MCM) loss leverages a masked captions prediction task to establish
detailed and generic relationships between textual and visual parts. The
proposed method is cost-effective and can easily retrieve specific persons with
textual captions. The architecture has only a dual-encoder without multi-level
branches or extra interaction modules, making a high-speed inference. Our
method achieves top results on three popular benchmarks with 73.81%, 74.25% and
57.35% Rank1 accuracy on the CUHK-PEDES, ICFG-PEDES, and RSTPReID,
respectively. We hope our scalable method will serve as a solid baseline and
help ease future research in TBPS. The code will be publicly available.
Related papers
- Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification [54.96876797812238]
We present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts.
The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images.
Experiment results across various datasets and models confirm CODER's effectiveness.
arXiv Detail & Related papers (2024-04-27T02:04:36Z) - Cross-Modal Retrieval Meets Inference:Improving Zero-Shot Classification
with Cross-Modal Retrieval [29.838375158101027]
Contrastive language-image pre-training (CLIP) has demonstrated remarkable zero-shot classification ability.
We propose X-MoRe, a novel inference method comprising two key steps: (1) cross-modal retrieval and (2) modal-confidence-based ensemble.
X-MoRe demonstrates robust performance across a diverse set of tasks without the need for additional training.
arXiv Detail & Related papers (2023-08-29T13:02:35Z) - PV2TEA: Patching Visual Modality to Textual-Established Information
Extraction [59.76117533540496]
We patch the visual modality to the textual-established attribute information extractor.
PV2TEA is an encoder-decoder architecture equipped with three bias reduction schemes.
Empirical results on real-world e-Commerce datasets demonstrate up to 11.74% absolute (20.97% relatively) F1 increase over unimodal baselines.
arXiv Detail & Related papers (2023-06-01T05:39:45Z) - Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image
Person Retrieval [29.884153827619915]
We present IRRA: a cross-modal Implicit Relation Reasoning and Aligning framework.
It learns relations between local visual-textual tokens and enhances global image-text matching.
The proposed method achieves new state-of-the-art results on all three public datasets.
arXiv Detail & Related papers (2023-03-22T12:11:59Z) - Towards Unifying Medical Vision-and-Language Pre-training via Soft
Prompts [63.84720380390935]
There exist two typical types, textiti.e., the fusion-encoder type and the dual-encoder type, depending on whether a heavy fusion module is used.
We propose an effective yet straightforward scheme named PTUnifier to unify the two types.
We first unify the input format by introducing visual and textual prompts, which serve as a feature bank that stores the most representative images/texts.
arXiv Detail & Related papers (2023-02-17T15:43:42Z) - FILIP: Fine-grained Interactive Language-Image Pre-Training [106.19474076935363]
Fine-grained Interactive Language-Image Pre-training achieves finer-level alignment through a cross-modal late interaction mechanism.
We construct a new large-scale image-text pair dataset called FILIP300M for pre-training.
Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks.
arXiv Detail & Related papers (2021-11-09T17:15:38Z) - Text-based Person Search in Full Images via Semantic-Driven Proposal
Generation [42.25611020956918]
We propose a new end-to-end learning framework which jointly optimize the pedestrian detection, identification and visual-semantic feature embedding tasks.
To take full advantage of the query text, the semantic features are leveraged to instruct the Region Proposal Network to pay more attention to the text-described proposals.
arXiv Detail & Related papers (2021-09-27T11:42:40Z) - Semantically Self-Aligned Network for Text-to-Image Part-aware Person
Re-identification [78.45528514468836]
Text-to-image person re-identification (ReID) aims to search for images containing a person of interest using textual descriptions.
We propose a Semantically Self-Aligned Network (SSAN) to handle the above problems.
To expedite future research in text-to-image ReID, we build a new database named ICFG-PEDES.
arXiv Detail & Related papers (2021-07-27T08:26:47Z) - Dual-path CNN with Max Gated block for Text-Based Person
Re-identification [6.1534388046236765]
A novel Dual-path CNN with Max Gated block (DCMG) is proposed to extract discriminative word embeddings.
The framework is based on two deep residual CNNs jointly optimized with cross-modal projection matching.
Our approach achieves the rank-1 score of 55.81% and outperforms the state-of-the-art method by 1.3%.
arXiv Detail & Related papers (2020-09-20T03:33:29Z) - Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using
Transformer Encoders [14.634046503477979]
We present a novel approach called Transformer Reasoning and Alignment Network (TERAN)
TERAN enforces a fine-grained match between the underlying components of images and sentences.
On the MS-COCO 1K test set, we obtain an improvement of 5.7% and 3.5% respectively on the image and the sentence retrieval tasks.
arXiv Detail & Related papers (2020-08-12T11:02:40Z) - A Novel Attention-based Aggregation Function to Combine Vision and
Language [55.7633883960205]
We propose a novel fully-attentive reduction method for vision and language.
Specifically, our approach computes a set of scores for each element of each modality employing a novel variant of cross-attention.
We test our approach on image-text matching and visual question answering, building fair comparisons with other reduction choices.
arXiv Detail & Related papers (2020-04-27T18:09:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.