Deep Multimodal Fusion for Generalizable Person Re-identification

• URL: http://arxiv.org/abs/2211.00933v1
• Date: Wed, 2 Nov 2022 07:42:48 GMT
• Title: Deep Multimodal Fusion for Generalizable Person Re-identification
• Authors: Suncheng Xiang, Hao Chen, Jingsheng Gao, Sijia Du, Jiawang Mou, Ting Liu, Dahong Qian, Yuzhuo Fu
• Abstract summary: DMF is a Deep Multimodal Fusion network for the general scenarios on person re-identification task. Rich semantic knowledge is introduced to assist in feature representation learning during the pre-training stage. A realistic dataset is adopted to fine-tine the pre-trained model for distribution alignment with real-world.
• Score: 15.250738959921872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Person re-identification plays a significant role in realistic scenarios due to its various applications in public security and video surveillance. Recently, leveraging the supervised or semi-unsupervised learning paradigms, which benefits from the large-scale datasets and strong computing performance, has achieved a competitive performance on a specific target domain. However, when Re-ID models are directly deployed in a new domain without target samples, they always suffer from considerable performance degradation and poor domain generalization. To address this challenge, in this paper, we propose DMF, a Deep Multimodal Fusion network for the general scenarios on person re-identification task, where rich semantic knowledge is introduced to assist in feature representation learning during the pre-training stage. On top of it, a multimodal fusion strategy is introduced to translate the data of different modalities into the same feature space, which can significantly boost generalization capability of Re-ID model. In the fine-tuning stage, a realistic dataset is adopted to fine-tine the pre-trained model for distribution alignment with real-world. Comprehensive experiments on benchmarks demonstrate that our proposed method can significantly outperform previous domain generalization or meta-learning methods. Our source code will also be publicly available at https://github.com/JeremyXSC/DMF.

Related papers

• Diverse Deep Feature Ensemble Learning for Omni-Domain Generalized Person Re-identification [30.208890289394994]
  Person ReID methods experience a significant drop in performance when trained and tested across different datasets. Our research reveals that domain generalization methods significantly underperform single-domain supervised methods on single dataset benchmarks. We propose a way to achieve ODG-ReID by creating deep feature diversity with self-ensembles.
  arXiv  Detail & Related papers  (2024-10-11T02:27:11Z)
• GM-DF: Generalized Multi-Scenario Deepfake Detection [49.072106087564144]
  Existing face forgery detection usually follows the paradigm of training models in a single domain. In this paper, we elaborately investigate the generalization capacity of deepfake detection models when jointly trained on multiple face forgery detection datasets.
  arXiv  Detail & Related papers  (2024-06-28T17:42:08Z)
• SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization [13.456240733175767]
  SimMMDG is a framework to overcome the challenges of achieving domain generalization in multi-modal scenarios. We employ supervised contrastive learning on the modality-shared features to ensure they possess joint properties and impose distance constraints. Our framework is theoretically well-supported and achieves strong performance in multi-modal DG on the EPIC-Kitchens dataset and the novel Human-Animal-Cartoon dataset.
  arXiv  Detail & Related papers  (2023-10-30T17:58:09Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Domain Generalization for Activity Recognition via Adaptive Feature Fusion [9.458837222079612]
  We propose emphAdaptive Feature Fusion for Activity Recognition(AFFAR). AFFAR learns to fuse the domain-invariant and domain-specific representations to improve the model's generalization performance. We apply AFAR to a real application, i.e., the diagnosis of Children's Attention Deficit Hyperactivity Disorder(ADHD)
  arXiv  Detail & Related papers  (2022-07-21T02:14:09Z)
• A Novel Mix-normalization Method for Generalizable Multi-source Person Re-identification [49.548815417844786]
  Person re-identification (Re-ID) has achieved great success in the supervised scenario. It is difficult to directly transfer the supervised model to arbitrary unseen domains due to the model overfitting to the seen source domains. We propose MixNorm, which consists of domain-aware mix-normalization (DMN) and domain-ware center regularization (DCR)
  arXiv  Detail & Related papers  (2022-01-24T18:09:38Z)
• Semi-Supervised Domain Generalizable Person Re-Identification [74.75528879336576]
  Existing person re-identification (re-id) methods are stuck when deployed to a new unseen scenario. Recent efforts have been devoted to domain adaptive person re-id where extensive unlabeled data in the new scenario are utilized in a transductive learning manner. We aim to explore multiple labeled datasets to learn generalized domain-invariant representations for person re-id.
  arXiv  Detail & Related papers  (2021-08-11T06:08:25Z)
• Learning to Generalize Unseen Domains via Memory-based Multi-Source Meta-Learning for Person Re-Identification [59.326456778057384]
  We propose the Memory-based Multi-Source Meta-Learning framework to train a generalizable model for unseen domains. We also present a meta batch normalization layer (MetaBN) to diversify meta-test features. Experiments demonstrate that our M$3$L can effectively enhance the generalization ability of the model for unseen domains.
  arXiv  Detail & Related papers  (2020-12-01T11:38:16Z)
• Multi-Domain Adversarial Feature Generalization for Person Re-Identification [52.835955258959785]
  We propose a multi-dataset feature generalization network (MMFA-AAE) It is capable of learning a universal domain-invariant feature representation from multiple labeled datasets and generalizing it to unseen' camera systems. It also surpasses many state-of-the-art supervised methods and unsupervised domain adaptation methods by a large margin.
  arXiv  Detail & Related papers  (2020-11-25T08:03:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.