Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection

• URL: http://arxiv.org/abs/2210.00753v1
• Date: Mon, 3 Oct 2022 08:10:12 GMT
• Title: Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection
• Authors: Xuanjun Chen, Haibin Wu, Helen Meng, Hung-yi Lee, Jyh-Shing Roger Jang
• Abstract summary: We reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks. We also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples.
• Score: 88.74863771919445
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.

Related papers

• Adversarial Robustness of Deep Reinforcement Learning based Dynamic Recommender Systems [50.758281304737444]
  We propose to explore adversarial examples and attack detection on reinforcement learning-based interactive recommendation systems. We first craft different types of adversarial examples by adding perturbations to the input and intervening on the casual factors. Then, we augment recommendation systems by detecting potential attacks with a deep learning-based classifier based on the crafted data.
  arXiv  Detail & Related papers  (2021-12-02T04:12:24Z)
• Towards A Conceptually Simple Defensive Approach for Few-shot classifiers Against Adversarial Support Samples [107.38834819682315]
  We study a conceptually simple approach to defend few-shot classifiers against adversarial attacks. We propose a simple attack-agnostic detection method, using the concept of self-similarity and filtering. Our evaluation on the miniImagenet (MI) and CUB datasets exhibit good attack detection performance.
  arXiv  Detail & Related papers  (2021-10-24T05:46:03Z)
• On the Exploitability of Audio Machine Learning Pipelines to Surreptitious Adversarial Examples [19.433014444284595]
  We introduce surreptitious adversarial examples, a new class of attacks that evades both human and pipeline controls. We show that this attack produces audio samples that are more surreptitious than previous attacks that aim solely for imperceptibility.
  arXiv  Detail & Related papers  (2021-08-03T16:21:08Z)
• Adversarial Visual Robustness by Causal Intervention [56.766342028800445]
  Adversarial training is the de facto most promising defense against adversarial examples. Yet, its passive nature inevitably prevents it from being immune to unknown attackers. We provide a causal viewpoint of adversarial vulnerability: the cause is the confounder ubiquitously existing in learning.
  arXiv  Detail & Related papers  (2021-06-17T14:23:54Z)
• Audio Attacks and Defenses against AED Systems - A Practical Study [2.365611283869544]
  We evaluate deep learning-based Audio Event Detection (AED) systems against evasion attacks through adversarial examples. We generate audio adversarial examples using two different types of noise, namely background and white noise, that can be used by the adversary to evade detection. We show that these countermeasures, when applied to audio input, can be successful.
  arXiv  Detail & Related papers  (2021-06-14T13:42:49Z)
• Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning [95.60856995067083]
  This work is among the first to perform adversarial defense for ASV without knowing the specific attack algorithms. We propose to perform adversarial defense from two perspectives: 1) adversarial perturbation purification and 2) adversarial perturbation detection. Experimental results show that our detection module effectively shields the ASV by detecting adversarial samples with an accuracy of around 80%.
  arXiv  Detail & Related papers  (2021-06-01T07:10:54Z)
• Can audio-visual integration strengthen robustness under multimodal attacks? [47.791552254215745]
  We use the audio-visual event recognition task against multimodal adversarial attacks as a proxy to investigate the robustness of audio-visual learning. We attack audio, visual, and both modalities to explore whether audio-visual integration still strengthens perception. For interpreting the multimodal interactions under attacks, we learn a weakly-supervised sound source visual localization model.
  arXiv  Detail & Related papers  (2021-04-05T16:46:45Z)
• Adversarial Attack and Defense Strategies for Deep Speaker Recognition Systems [44.305353565981015]
  This paper considers several state-of-the-art adversarial attacks to a deep speaker recognition system, employing strong defense methods as countermeasures. Experiments show that the speaker recognition systems are vulnerable to adversarial attacks, and the strongest attacks can reduce the accuracy of the system from 94% to even 0%.
  arXiv  Detail & Related papers  (2020-08-18T00:58:19Z)
• Defense for Black-box Attacks on Anti-spoofing Models by Self-Supervised Learning [71.17774313301753]
  We explore the robustness of self-supervised learned high-level representations by using them in the defense against adversarial attacks. Experimental results on the ASVspoof 2019 dataset demonstrate that high-level representations extracted by Mockingjay can prevent the transferability of adversarial examples.
  arXiv  Detail & Related papers  (2020-06-05T03:03:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.