CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification
without Concrete Text Labels
- URL: http://arxiv.org/abs/2211.13977v2
- Date: Tue, 29 Nov 2022 13:30:17 GMT
- Title: CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification
without Concrete Text Labels
- Authors: Siyuan Li, Li Sun, Qingli Li
- Abstract summary: We propose a two-stage strategy to facilitate a better visual representation in image re-identification tasks.
The key idea is to fully exploit the cross-modal description ability in CLIP through a set of learnable text tokens for each ID.
The effectiveness of the proposed strategy is validated on several datasets for the person or vehicle ReID tasks.
- Score: 28.42405456691034
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Pre-trained vision-language models like CLIP have recently shown superior
performances on various downstream tasks, including image classification and
segmentation. However, in fine-grained image re-identification (ReID), the
labels are indexes, lacking concrete text descriptions. Therefore, it remains
to be determined how such models could be applied to these tasks. This paper
first finds out that simply fine-tuning the visual model initialized by the
image encoder in CLIP, has already obtained competitive performances in various
ReID tasks. Then we propose a two-stage strategy to facilitate a better visual
representation. The key idea is to fully exploit the cross-modal description
ability in CLIP through a set of learnable text tokens for each ID and give
them to the text encoder to form ambiguous descriptions. In the first training
stage, image and text encoders from CLIP keep fixed, and only the text tokens
are optimized from scratch by the contrastive loss computed within a batch. In
the second stage, the ID-specific text tokens and their encoder become static,
providing constraints for fine-tuning the image encoder. With the help of the
designed loss in the downstream task, the image encoder is able to represent
data as vectors in the feature embedding accurately. The effectiveness of the
proposed strategy is validated on several datasets for the person or vehicle
ReID tasks. Code is available at https://github.com/Syliz517/CLIP-ReID.
Related papers
- Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification [54.96876797812238]
We present a novel CrOss-moDal nEighbor Representation(CODER) based on the distance structure between images and their neighbor texts.
The key to construct a high-quality CODER lies in how to create a vast amount of high-quality and diverse texts to match with images.
Experiment results across various datasets and models confirm CODER's effectiveness.
arXiv Detail & Related papers (2024-04-27T02:04:36Z) - Text encoders bottleneck compositionality in contrastive vision-language
models [76.2406963762722]
We train text-only recovery probes that aim to reconstruct captions from single-vector text representations.
We find that CLIP's text encoder falls short on more compositional inputs.
Results suggest text-only recoverability is a necessary (but not sufficient) condition for modeling compositional factors.
arXiv Detail & Related papers (2023-05-24T08:48:44Z) - DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only
Training [73.74291217502928]
We propose a simple framework, named DeCap, for zero-shot captioning.
We introduce a lightweight visual-aware language decoder.
We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input.
arXiv Detail & Related papers (2023-03-06T11:02:47Z) - Texts as Images in Prompt Tuning for Multi-Label Image Recognition [70.9310322461598]
We advocate that image-text contrastive learning makes it feasible to treat texts as images for prompt tuning and introduce TaI prompting.
Particularly, we apply TaI prompting to multi-label image recognition, where sentences in the wild serve as alternatives to images for prompt tuning.
Our proposed TaI-DPT outperforms zero-shot CLIP by a large margin on multiple benchmarks.
arXiv Detail & Related papers (2022-11-23T07:00:11Z) - Fine-grained Image Captioning with CLIP Reward [104.71533106301598]
We propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function.
We also propose a simple finetuning strategy of the CLIP text encoder to improve grammar that does not require extra text annotation.
In experiments on text-to-image retrieval and FineCapEval, the proposed CLIP-guided model generates more distinctive captions than the CIDEr-optimized model.
arXiv Detail & Related papers (2022-05-26T02:46:09Z) - CoCa: Contrastive Captioners are Image-Text Foundation Models [41.759438751996505]
Contrastive Captioner (CoCa) is a minimalist design to pretrain an image-text encoder-decoder foundation model.
By sharing the same computational graph, the two training objectives are computed efficiently with minimal overhead.
CoCa achieves state-of-the-art performance with zero-shot transfer or minimal task-specific adaptation on a broad range of downstream tasks.
arXiv Detail & Related papers (2022-05-04T07:01:14Z) - CRIS: CLIP-Driven Referring Image Segmentation [71.56466057776086]
We propose an end-to-end CLIP-Driven Referring Image framework (CRIS)
CRIS resorts to vision-language decoding and contrastive learning for achieving the text-to-pixel alignment.
Our proposed framework significantly outperforms the state-of-the-art performance without any post-processing.
arXiv Detail & Related papers (2021-11-30T07:29:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.