Audio-Visual Event Recognition through the lens of Adversary
- URL: http://arxiv.org/abs/2011.07430v1
- Date: Sun, 15 Nov 2020 01:36:09 GMT
- Title: Audio-Visual Event Recognition through the lens of Adversary
- Authors: Juncheng B Li, Kaixin Ma, Shuhui Qu, Po-Yao Huang, Florian Metze
- Abstract summary: This work aims to study several key questions related to multimodal learning through the lens of adversarial noises.
We construct adversarial examples to attack state-of-the-art neural models trained on Google AudioSet.
- Score: 38.497204362659296
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: As audio/visual classification models are widely deployed for sensitive tasks
like content filtering at scale, it is critical to understand their robustness
along with improving the accuracy. This work aims to study several key
questions related to multimodal learning through the lens of adversarial
noises: 1) The trade-off between early/middle/late fusion affecting its
robustness and accuracy 2) How do different frequency/time domain features
contribute to the robustness? 3) How do different neural modules contribute to
the adversarial noise? In our experiment, we construct adversarial examples to
attack state-of-the-art neural models trained on Google AudioSet. We compare
how much attack potency in terms of adversarial perturbation of size $\epsilon$
using different $L_p$ norms we would need to "deactivate" the victim model.
Using adversarial noise to ablate multimodal models, we are able to provide
insights into what is the best potential fusion strategy to balance the model
parameters/accuracy and robustness trade-off and distinguish the robust
features versus the non-robust features that various neural networks model tend
to learn.
Related papers
- Adversarial Attacks and Dimensionality in Text Classifiers [3.4179091429029382]
Adversarial attacks on machine learning algorithms have been a key deterrent to the adoption of AI in many real-world use cases.
We study adversarial examples in the field of natural language processing, specifically text classification tasks.
arXiv Detail & Related papers (2024-04-03T11:49:43Z) - Improving the Robustness of Summarization Systems with Dual Augmentation [68.53139002203118]
A robust summarization system should be able to capture the gist of the document, regardless of the specific word choices or noise in the input.
We first explore the summarization models' robustness against perturbations including word-level synonym substitution and noise.
We propose a SummAttacker, which is an efficient approach to generating adversarial samples based on language models.
arXiv Detail & Related papers (2023-06-01T19:04:17Z) - Wavelets Beat Monkeys at Adversarial Robustness [0.8702432681310401]
We show how physically inspired structures yield new insights into robustness that were previously only thought possible by meticulously mimicking the human cortex.
Our work shows how physically inspired structures yield new insights into robustness that were previously only thought possible by meticulously mimicking the human cortex.
arXiv Detail & Related papers (2023-04-19T03:41:30Z) - Exploring Adversarial Attacks on Neural Networks: An Explainable
Approach [18.063187159491182]
We analyze the response characteristics of the VGG-16 model when the input images are mixed with adversarial noise and statistically similar Gaussian random noise.
Our work could provide valuable insights into developing more reliable Deep Neural Network (DNN) models.
arXiv Detail & Related papers (2023-03-08T07:59:44Z) - Multitasking Models are Robust to Structural Failure: A Neural Model for
Bilingual Cognitive Reserve [78.3500985535601]
We find a surprising connection between multitask learning and robustness to neuron failures.
Our experiments show that bilingual language models retain higher performance under various neuron perturbations.
We provide a theoretical justification for this robustness by mathematically analyzing linear representation learning.
arXiv Detail & Related papers (2022-10-20T22:23:27Z) - Analyzing Robustness of End-to-End Neural Models for Automatic Speech
Recognition [11.489161072526677]
We investigate robustness properties of pre-trained neural models for automatic speech recognition.
In this work, we perform a robustness analysis of the pre-trained neural models wav2vec2, HuBERT and DistilHuBERT on the LibriSpeech and TIMIT datasets.
arXiv Detail & Related papers (2022-08-17T20:00:54Z) - Characterizing and overcoming the greedy nature of learning in
multi-modal deep neural networks [62.48782506095565]
We show that due to the greedy nature of learning in deep neural networks, models tend to rely on just one modality while under-fitting the other modalities.
We propose an algorithm to balance the conditional learning speeds between modalities during training and demonstrate that it indeed addresses the issue of greedy learning.
arXiv Detail & Related papers (2022-02-10T20:11:21Z) - Firearm Detection via Convolutional Neural Networks: Comparing a
Semantic Segmentation Model Against End-to-End Solutions [68.8204255655161]
Threat detection of weapons and aggressive behavior from live video can be used for rapid detection and prevention of potentially deadly incidents.
One way for achieving this is through the use of artificial intelligence and, in particular, machine learning for image analysis.
We compare a traditional monolithic end-to-end deep learning model and a previously proposed model based on an ensemble of simpler neural networks detecting fire-weapons via semantic segmentation.
arXiv Detail & Related papers (2020-12-17T15:19:29Z) - Learning to Learn from Mistakes: Robust Optimization for Adversarial
Noise [1.976652238476722]
We train a meta-optimizer which learns to robustly optimize a model using adversarial examples and is able to transfer the knowledge learned to new models.
Experimental results show the meta-optimizer is consistent across different architectures and data sets, suggesting it is possible to automatically patch adversarial vulnerabilities.
arXiv Detail & Related papers (2020-08-12T11:44:01Z) - Learning to Generate Noise for Multi-Attack Robustness [126.23656251512762]
Adversarial learning has emerged as one of the successful techniques to circumvent the susceptibility of existing methods against adversarial perturbations.
In safety-critical applications, this makes these methods extraneous as the attacker can adopt diverse adversaries to deceive the system.
We propose a novel meta-learning framework that explicitly learns to generate noise to improve the model's robustness against multiple types of attacks.
arXiv Detail & Related papers (2020-06-22T10:44:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.