ChatReID: Open-ended Interactive Person Retrieval via Hierarchical Progressive Tuning for Vision Language Models

• URL: http://arxiv.org/abs/2502.19958v2
• Date: Sat, 22 Mar 2025 11:13:15 GMT
• Title: ChatReID: Open-ended Interactive Person Retrieval via Hierarchical Progressive Tuning for Vision Language Models
• Authors: Ke Niu, Haiyang Yu, Mengyang Zhao, Teng Fu, Siyang Yi, Wei Lu, Bin Li, Xuelin Qian, Xiangyang Xue,
• Abstract summary: Person re-identification is a crucial task in computer vision, aiming to recognize individuals across non-overlapping camera views.<n>We propose a novel framework ChatReID, that shifts the focus towards a text-side-dominated retrieval paradigm, enabling flexible and interactive re-identification.<n>We introduce a hierarchical progressive tuning strategy, which endows Re-ID ability through three stages of tuning, i.e., from person attribute understanding to fine-grained image retrieval and to multi-modal task reasoning.
• Score: 49.09606704563898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Person re-identification (Re-ID) is a crucial task in computer vision, aiming to recognize individuals across non-overlapping camera views. While recent advanced vision-language models (VLMs) excel in logical reasoning and multi-task generalization, their applications in Re-ID tasks remain limited. They either struggle to perform accurate matching based on identity-relevant features or assist image-dominated branches as auxiliary semantics. In this paper, we propose a novel framework ChatReID, that shifts the focus towards a text-side-dominated retrieval paradigm, enabling flexible and interactive re-identification. To integrate the reasoning abilities of language models into Re-ID pipelines, We first present a large-scale instruction dataset, which contains more than 8 million prompts to promote the model fine-tuning. Next. we introduce a hierarchical progressive tuning strategy, which endows Re-ID ability through three stages of tuning, i.e., from person attribute understanding to fine-grained image retrieval and to multi-modal task reasoning. Extensive experiments across ten popular benchmarks demonstrate that ChatReID outperforms existing methods, achieving state-of-the-art performance in all Re-ID tasks. More experiments demonstrate that ChatReID not only has the ability to recognize fine-grained details but also to integrate them into a coherent reasoning process.

Related papers

• SemCORE: A Semantic-Enhanced Generative Cross-Modal Retrieval Framework with MLLMs [70.79124435220695]
  We propose a novel unified Semantic-enhanced generative Cross-mOdal REtrieval framework (SemCORE) We first construct a Structured natural language IDentifier (SID) that effectively aligns target identifiers with generative models optimized for natural language comprehension and generation. We then introduce a Generative Semantic Verification (GSV) strategy enabling fine-grained target discrimination.
  arXiv  Detail & Related papers  (2025-04-17T17:59:27Z)
• ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning [62.61187785810336]
  ImageScope is a training-free, three-stage framework that unifies language-guided image retrieval tasks. In the first stage, we improve the robustness of the framework by synthesizing search intents across varying levels of semantic granularity. In the second and third stages, we reflect on retrieval results by verifying predicate propositions locally, and performing pairwise evaluations globally.
  arXiv  Detail & Related papers  (2025-03-13T08:43:24Z)
• HEISIR: Hierarchical Expansion of Inverted Semantic Indexing for Training-free Retrieval of Conversational Data using LLMs [0.3277163122167434]
  This paper introduces HEISIR, a novel framework that enhances semantic understanding in conversational data retrieval. Heisir implements a two-step process: (1) Hierarchical Triplets Formulation and (2) Adjunct Augmentation, creating semantic indices consisting of Subject-Verb-Object-Adjunct (SVOA) quadruplets. Our experimental results demonstrate that HEISIR outperforms fine-tuned models across various embedding types and language models.
  arXiv  Detail & Related papers  (2025-03-06T06:39:25Z)
• Enhancing Visual Representation for Text-based Person Searching [9.601697802095119]
  VFE-TPS is a Visual Feature Enhanced Text-based Person Search model. It introduces a pre-trained backbone CLIP to learn basic multimodal features. It constructs Text Guided Masked Image Modeling task to enhance the model's ability of learning local visual details.
  arXiv  Detail & Related papers  (2024-12-30T01:38:14Z)
• All in One Framework for Multimodal Re-identification in the Wild [58.380708329455466]
  multimodal learning paradigm for ReID introduced, referred to as All-in-One (AIO) AIO harnesses a frozen pre-trained big model as an encoder, enabling effective multimodal retrieval without additional fine-tuning. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts.
  arXiv  Detail & Related papers  (2024-05-08T01:04:36Z)
• Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification [64.36210786350568]
  We propose a novel learning framework named textbfEDITOR to select diverse tokens from vision Transformers for multi-modal object ReID. Our framework can generate more discriminative features for multi-modal object ReID.
  arXiv  Detail & Related papers  (2024-03-15T12:44:35Z)
• Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond [99.73306923465424]
  We introduce a generative cross-modal retrieval framework, which assigns unique identifier strings to represent images. By memorizing images in MLLMs, we introduce a new paradigm to cross-modal retrieval, distinct from previous discriminative approaches.
  arXiv  Detail & Related papers  (2024-02-16T16:31:46Z)
• Multi-Prompts Learning with Cross-Modal Alignment for Attribute-based Person Re-Identification [18.01407937934588]
  We present a new framework called Multi-Prompts ReID (MP-ReID) based on prompt learning and language models. MP-ReID learns to hallucinate diverse, informative, and promptable sentences for describing the query images. Explicit prompts are obtained by ensembling generation models, such as ChatGPT and VQA models.
  arXiv  Detail & Related papers  (2023-12-28T03:00:19Z)
• VILLS -- Video-Image Learning to Learn Semantics for Person Re-Identification [51.89551385538251]
  We propose VILLS (Video-Image Learning to Learn Semantics), a self-supervised method that jointly learns spatial and temporal features from images and videos. VILLS first designs a local semantic extraction module that adaptively extracts semantically consistent and robust spatial features. Then, VILLS designs a unified feature learning and adaptation module to represent image and video modalities in a consistent feature space.
  arXiv  Detail & Related papers  (2023-11-27T19:30:30Z)
• Learning Transferable Pedestrian Representation from Multimodal Information Supervision [174.5150760804929]
  VAL-PAT is a novel framework that learns transferable representations to enhance various pedestrian analysis tasks with multimodal information. We first perform pre-training on LUPerson-TA dataset, where each image contains text and attribute annotations. We then transfer the learned representations to various downstream tasks, including person reID, person attribute recognition and text-based person search.
  arXiv  Detail & Related papers  (2023-04-12T01:20:58Z)
• Learning Progressive Modality-shared Transformers for Effective Visible-Infrared Person Re-identification [27.75907274034702]
  We propose a novel deep learning framework named Progressive Modality-shared Transformer (PMT) for effective VI-ReID. To reduce the negative effect of modality gaps, we first take the gray-scale images as an auxiliary modality and propose a progressive learning strategy. To cope with the problem of large intra-class differences and small inter-class differences, we propose a Discriminative Center Loss.
  arXiv  Detail & Related papers  (2022-12-01T02:20:16Z)
• JARVIS: A Neuro-Symbolic Commonsense Reasoning Framework for Conversational Embodied Agents [14.70666899147632]
  We propose a Neuro-Symbolic Commonsense Reasoning framework for modular, generalizable, and interpretable conversational embodied agents. Our framework achieves state-of-the-art (SOTA) results on all three dialog-based embodied tasks, including Execution from Dialog History (EDH), Trajectory from Dialog (TfD), and Two-Agent Task Completion (TATC) Our model ranks first in the Alexa Prize SimBot Public Benchmark Challenge.
  arXiv  Detail & Related papers  (2022-08-28T18:30:46Z)
• On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization [89.94078728495423]
  We show that recent advances in each modality, CLIP image representations and scaling of language models, do not consistently improve multimodal self-rationalization of tasks with multimodal inputs. Our findings call for a backbone modelling approach that can be built on to advance text generation from images and text beyond image captioning.
  arXiv  Detail & Related papers  (2022-05-24T00:52:40Z)
• Semantic Consistency and Identity Mapping Multi-Component Generative Adversarial Network for Person Re-Identification [39.605062525247135]
  We propose a semantic consistency and identity mapping multi-component generative adversarial network (SC-IMGAN) which provides style adaptation from one to many domains. Our proposed method outperforms state-of-the-art techniques on six challenging person Re-ID datasets.
  arXiv  Detail & Related papers  (2021-04-28T14:12:29Z)
• Cross-Resolution Adversarial Dual Network for Person Re-Identification and Beyond [59.149653740463435]
  Person re-identification (re-ID) aims at matching images of the same person across camera views. Due to varying distances between cameras and persons of interest, resolution mismatch can be expected. We propose a novel generative adversarial network to address cross-resolution person re-ID.
  arXiv  Detail & Related papers  (2020-02-19T07:21:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.