PACOL: Poisoning Attacks Against Continual Learners
- URL: http://arxiv.org/abs/2311.10919v1
- Date: Sat, 18 Nov 2023 00:20:57 GMT
- Title: PACOL: Poisoning Attacks Against Continual Learners
- Authors: Huayu Li and Gregory Ditzler
- Abstract summary: In this work, we demonstrate that continual learning systems can be manipulated by malicious misinformation.
We present a new category of data poisoning attacks specific for continual learners, which we refer to as em Poisoning Attacks Against Continual learners (PACOL)
A comprehensive set of experiments shows the vulnerability of commonly used generative replay and regularization-based continual learning approaches against attack methods.
- Score: 1.569413950416037
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Continual learning algorithms are typically exposed to untrusted sources that
contain training data inserted by adversaries and bad actors. An adversary can
insert a small number of poisoned samples, such as mislabeled samples from
previously learned tasks, or intentional adversarial perturbed samples, into
the training datasets, which can drastically reduce the model's performance. In
this work, we demonstrate that continual learning systems can be manipulated by
malicious misinformation and present a new category of data poisoning attacks
specific for continual learners, which we refer to as {\em Poisoning Attacks
Against Continual Learners} (PACOL). The effectiveness of labeling flipping
attacks inspires PACOL; however, PACOL produces attack samples that do not
change the sample's label and produce an attack that causes catastrophic
forgetting. A comprehensive set of experiments shows the vulnerability of
commonly used generative replay and regularization-based continual learning
approaches against attack methods. We evaluate the ability of label-flipping
and a new adversarial poison attack, namely PACOL proposed in this work, to
force the continual learning system to forget the knowledge of a learned
task(s). More specifically, we compared the performance degradation of
continual learning systems trained on benchmark data streams with and without
poisoning attacks. Moreover, we discuss the stealthiness of the attacks in
which we test the success rate of data sanitization defense and other outlier
detection-based defenses for filtering out adversarial samples.
Related papers
- Wicked Oddities: Selectively Poisoning for Effective Clean-Label Backdoor Attacks [11.390175856652856]
Clean-label attacks are a more stealthy form of backdoor attacks that can perform the attack without changing the labels of poisoned data.
We study different strategies for selectively poisoning a small set of training samples in the target class to boost the attack success rate.
Our threat model poses a serious threat in training machine learning models with third-party datasets.
arXiv Detail & Related papers (2024-07-15T15:38:21Z) - SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks [53.28390057407576]
Modern NLP models are often trained on public datasets drawn from diverse sources.
Data poisoning attacks can manipulate the model's behavior in ways engineered by the attacker.
Several strategies have been proposed to mitigate the risks associated with backdoor attacks.
arXiv Detail & Related papers (2024-05-19T14:50:09Z) - FreqFed: A Frequency Analysis-Based Approach for Mitigating Poisoning
Attacks in Federated Learning [98.43475653490219]
Federated learning (FL) is susceptible to poisoning attacks.
FreqFed is a novel aggregation mechanism that transforms the model updates into the frequency domain.
We demonstrate that FreqFed can mitigate poisoning attacks effectively with a negligible impact on the utility of the aggregated model.
arXiv Detail & Related papers (2023-12-07T16:56:24Z) - Transferable Availability Poisoning Attacks [23.241524904589326]
We consider availability data poisoning attacks, where an adversary aims to degrade the overall test accuracy of a machine learning model.
Existing poisoning strategies can achieve the attack goal but assume the victim to employ the same learning method as what the adversary uses to mount the attack.
We propose Transferable Poisoning, which first leverages the intrinsic characteristics of alignment and uniformity to enable better unlearnability.
arXiv Detail & Related papers (2023-10-08T12:22:50Z) - Exploring Model Dynamics for Accumulative Poisoning Discovery [62.08553134316483]
We propose a novel information measure, namely, Memorization Discrepancy, to explore the defense via the model-level information.
By implicitly transferring the changes in the data manipulation to that in the model outputs, Memorization Discrepancy can discover the imperceptible poison samples.
We thoroughly explore its properties and propose Discrepancy-aware Sample Correction (DSC) to defend against accumulative poisoning attacks.
arXiv Detail & Related papers (2023-06-06T14:45:24Z) - Rethinking Backdoor Data Poisoning Attacks in the Context of
Semi-Supervised Learning [5.417264344115724]
Semi-supervised learning methods can train high-accuracy machine learning models with a fraction of the labeled training samples required for traditional supervised learning.
Such methods do not typically involve close review of the unlabeled training samples, making them tempting targets for data poisoning attacks.
We show that simple poisoning attacks that influence the distribution of the poisoned samples' predicted labels are highly effective.
arXiv Detail & Related papers (2022-12-05T20:21:31Z) - Amplifying Membership Exposure via Data Poisoning [18.799570863203858]
In this paper, we investigate the third type of exploitation of data poisoning - increasing the risks of privacy leakage of benign training samples.
We propose a set of data poisoning attacks to amplify the membership exposure of the targeted class.
Our results show that the proposed attacks can substantially increase the membership inference precision with minimum overall test-time model performance degradation.
arXiv Detail & Related papers (2022-11-01T13:52:25Z) - Adversarial Robustness of Deep Reinforcement Learning based Dynamic
Recommender Systems [50.758281304737444]
We propose to explore adversarial examples and attack detection on reinforcement learning-based interactive recommendation systems.
We first craft different types of adversarial examples by adding perturbations to the input and intervening on the casual factors.
Then, we augment recommendation systems by detecting potential attacks with a deep learning-based classifier based on the crafted data.
arXiv Detail & Related papers (2021-12-02T04:12:24Z) - Accumulative Poisoning Attacks on Real-time Data [56.96241557830253]
We show that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects.
Our work validates that a well-designed but straightforward attacking strategy can dramatically amplify the poisoning effects.
arXiv Detail & Related papers (2021-06-18T08:29:53Z) - Learning and Certification under Instance-targeted Poisoning [49.55596073963654]
We study PAC learnability and certification under instance-targeted poisoning attacks.
We show that when the budget of the adversary scales sublinearly with the sample complexity, PAC learnability and certification are achievable.
We empirically study the robustness of K nearest neighbour, logistic regression, multi-layer perceptron, and convolutional neural network on real data sets.
arXiv Detail & Related papers (2021-05-18T17:48:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.