Modality Unified Attack for Omni-Modality Person Re-Identification

• URL: http://arxiv.org/abs/2501.12761v1
• Date: Wed, 22 Jan 2025 09:54:43 GMT
• Title: Modality Unified Attack for Omni-Modality Person Re-Identification
• Authors: Yuan Bian, Min Liu, Yunqi Yi, Xueping Wang, Yunfeng Ma, Yaonan Wang,
• Abstract summary: We propose a novel Modality Unified Attack method to train adversarial generators to attack different omni-modality models. Experiments show that our method can effectively attack the omni-modality re-id models, achieving 55.9%, 24.4%, 49.0% and 62.7% mean mAP Drop Rate.
• Score: 16.624135145315673
• License:
• Abstract: Deep learning based person re-identification (re-id) models have been widely employed in surveillance systems. Recent studies have demonstrated that black-box single-modality and cross-modality re-id models are vulnerable to adversarial examples (AEs), leaving the robustness of multi-modality re-id models unexplored. Due to the lack of knowledge about the specific type of model deployed in the target black-box surveillance system, we aim to generate modality unified AEs for omni-modality (single-, cross- and multi-modality) re-id models. Specifically, we propose a novel Modality Unified Attack method to train modality-specific adversarial generators to generate AEs that effectively attack different omni-modality models. A multi-modality model is adopted as the surrogate model, wherein the features of each modality are perturbed by metric disruption loss before fusion. To collapse the common features of omni-modality models, Cross Modality Simulated Disruption approach is introduced to mimic the cross-modality feature embeddings by intentionally feeding images to non-corresponding modality-specific subnetworks of the surrogate model. Moreover, Multi Modality Collaborative Disruption strategy is devised to facilitate the attacker to comprehensively corrupt the informative content of person images by leveraging a multi modality feature collaborative metric disruption loss. Extensive experiments show that our MUA method can effectively attack the omni-modality re-id models, achieving 55.9%, 24.4%, 49.0% and 62.7% mean mAP Drop Rate, respectively.

Related papers

• MAA: Meticulous Adversarial Attack against Vision-Language Pre-trained Models [30.04163729936878]
  Meticulous Adrial Attack (MAA) fully exploit model-independent characteristics and vulnerabilities of individual samples. MAA emphasizes fine-grained optimization of adversarial images by developing a novel resizing and sliding crop (RScrop) technique.
  arXiv  Detail & Related papers  (2025-02-12T02:53:27Z)
• Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models [6.610033827647869]
  In real-world scenarios, consistently acquiring complete multimodal data presents significant challenges. This often leads to the issue of missing modalities, where data for certain modalities are absent. We propose a novel framework integrating parameter-efficient fine-tuning of unimodal pretrained models with a self-supervised joint-embedding learning method.
  arXiv  Detail & Related papers  (2024-07-17T14:44:25Z)
• Dealing with All-stage Missing Modality: Towards A Universal Model with Robust Reconstruction and Personalization [14.606035444283984]
  Current approaches focus on developing models that handle modality-incomplete inputs during inference. We propose a robust universal model with modality reconstruction and model personalization. Our method has been extensively validated on two brain tumor segmentation benchmarks.
  arXiv  Detail & Related papers  (2024-06-04T06:07:24Z)
• All in One Framework for Multimodal Re-identification in the Wild [58.380708329455466]
  multimodal learning paradigm for ReID introduced, referred to as All-in-One (AIO) AIO harnesses a frozen pre-trained big model as an encoder, enabling effective multimodal retrieval without additional fine-tuning. Experiments on cross-modal and multimodal ReID reveal that AIO not only adeptly handles various modal data but also excels in challenging contexts.
  arXiv  Detail & Related papers  (2024-05-08T01:04:36Z)
• MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models [34.802736332993994]
  We propose MMCert, the first certified defense against adversarial attacks to a multi-modal model. We evaluate our MMCert using two benchmark datasets: one for the multi-modal road segmentation task and the other for the multi-modal emotion recognition task.
  arXiv  Detail & Related papers  (2024-03-28T01:05:06Z)
• Model Stealing Attack against Graph Classification with Authenticity, Uncertainty and Diversity [80.16488817177182]
  GNNs are vulnerable to the model stealing attack, a nefarious endeavor geared towards duplicating the target model via query permissions. We introduce three model stealing attacks to adapt to different actual scenarios.
  arXiv  Detail & Related papers  (2023-12-18T05:42:31Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Method [115.29382166356478]
  We introduce the adversarial retrieval attack (AREA) task. It is meant to trick DR models into retrieving a target document that is outside the initial set of candidate documents retrieved by the DR model. We find that the promising results that have previously been reported on attacking NRMs, do not generalize to DR models. We propose to formalize attacks on DR models as a contrastive learning problem in a multi-view representation space.
  arXiv  Detail & Related papers  (2023-08-19T00:24:59Z)
• Ensemble Modeling for Multimodal Visual Action Recognition [50.38638300332429]
  We propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset.
  arXiv  Detail & Related papers  (2023-08-10T08:43:20Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• Defending Multimodal Fusion Models against Single-Source Adversaries [6.019777076722421]
  We show that standard multimodal fusion models are vulnerable to single-source adversaries. An attack on any single modality can overcome the correct information from multiple unperturbed modalities and cause the model to fail. Motivated by this finding, we propose an adversarially robust fusion strategy.
  arXiv  Detail & Related papers  (2022-06-25T18:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.