Generative Retrieval for Unsupervised Text-Based Person Search
Abstract Overview
This paper studies unsupervised text-based person search (TBPS), where model training relies solely on unlabeled person images without human-annotated image-text pairs. It proposes GTR+, a two-stage generation-then-retrieval framework: first, pseudo textual descriptions are generated via a three-tier process that extracts basic attributes, refines fine-grained details through hard negatives, and enriches stylistic diversity; then, a retrieval model is trained using an adaptive confidence-weighting scheme to mitigate the effect of noisy generated texts. The paper also introduces LargeFine-Person, a large-scale pre-training dataset of 1.2 million images and 3.6 million automatically generated descriptions sourced from LUPerson and LPW. Evaluations on CUHK-PEDES, ICFG-PEDES, and RSTPReid show that GTR+ outperforms existing unsupervised TBPS methods and gains further performance boosts when pre-trained on LargeFine-Person.
Novelty
The primary novelty lies in formulating unsupervised TBPS via a two-stage generative retrieval pipeline that integrates tiered pseudo-text generation (question-and-answer, inter-sample contrastive, and stylized expansion) with dynamic GMM-calibrated confidence weighting during retrieval training. The paper also introduces LargeFine-Person, an unannotated 3.6-million-pair pre-training resource offering fine-grained and stylistically diverse descriptions.
Results
In the unsupervised setting, GTR+ achieves state-of-the-art performance with Rank-1/mAP scores of 61.35/55.75% on CUHK-PEDES, 47.81/28.75% on ICFG-PEDES, and 54.75/43.79% on RSTPReid. When pre-trained on LargeFine-Person, these metrics increase to 64.65/58.67%, 52.78/33.99%, and 55.70/43.86%, respectively. Ablations confirm that each generation tier and the adaptive confidence weighting mechanism provide measurable performance gains over baseline and masking strategies.
Key Points
- GTR+ generates synthetic person descriptions through three sequential tiers: an automated Q&A mechanism for visual attributes, an inter-sample contrastive mechanism with hard negatives for fine details, and a stylized expansion mechanism for sentence diversity.
- The retrieval learning stage uses a two-component Gaussian Mixture Model combining real-time image-text similarity and prior text generation probability to adaptively down-weight noisy pairs during training.
- LargeFine-Person contains 1.2 million denoised images and 3.6 million generated image-text pairs, establishing an effective pre-training benchmark for unsupervised text-based person search.