One-Shot Machine Unlearning with Mnemonic Code
- URL: http://arxiv.org/abs/2306.05670v1
- Date: Fri, 9 Jun 2023 04:59:24 GMT
- Title: One-Shot Machine Unlearning with Mnemonic Code
- Authors: Tomoya Yamashita and Masanori Yamada and Takashi Shibata
- Abstract summary: Machine unlearning (MU) aims at forgetting about undesirable training data from a trained deep learning model.
A naive MU approach is to re-train the whole model with the training data from which the undesirable data has been removed.
We propose a one-shot MU method, which does not need additional training.
- Score: 5.579745503613096
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep learning has achieved significant improvements in accuracy and has been
applied to various fields. With the spread of deep learning, a new problem has
also emerged; deep learning models can sometimes have undesirable information
from an ethical standpoint. This problem must be resolved if deep learning is
to make sensitive decisions such as hiring and prison sentencing. Machine
unlearning (MU) is the research area that responds to such demands. MU aims at
forgetting about undesirable training data from a trained deep learning model.
A naive MU approach is to re-train the whole model with the training data from
which the undesirable data has been removed. However, re-training the whole
model can take a huge amount of time and consumes significant computer
resources. To make MU even more practical, a simple-yet-effective MU method is
required. In this paper, we propose a one-shot MU method, which does not need
additional training. To design one-shot MU, we add noise to the model
parameters that are sensitive to undesirable information. In our proposed
method, we use the Fisher information matrix (FIM) to estimate the sensitive
model parameters. Training data were usually used to evaluate the FIM in
existing methods. In contrast, we avoid the need to retain the training data
for calculating the FIM by using class-specific synthetic signals called
mnemonic code. Extensive experiments using artificial and natural datasets
demonstrate that our method outperforms the existing methods.
Related papers
- An Information Theoretic Metric for Evaluating Unlearning Models [20.143627174765985]
Machine unlearning (MU) addresses privacy concerns by removing information of forgetting data' samples from trained models.
We propose a metric that quantifies the residual information about forgetting data samples in intermediate features using mutual information.
arXiv Detail & Related papers (2024-05-28T06:57:01Z) - Towards Natural Machine Unlearning [22.49633264030417]
Machine unlearning (MU) aims to eliminate information that has been learned from specific training data, namely forgetting data, from a pre-trained model.
Currently, the mainstream of existing MU methods involves modifying the forgetting data with incorrect labels and subsequently fine-tuning the model.
We introduce textitnatural machine unlearning, we inject correct information from the remaining data to the forgetting samples when changing their labels.
Through pairing these adjusted samples with their labels, the model will tend to use the injected correct information and naturally suppress the information meant to be forgotten.
arXiv Detail & Related papers (2024-05-24T12:23:38Z) - Unlearnable Algorithms for In-context Learning [36.895152458323764]
In this paper, we focus on efficient unlearning methods for the task adaptation phase of a pretrained large language model.
We observe that an LLM's ability to do in-context learning for task adaptation allows for efficient exact unlearning of task adaptation training data.
We propose a new holistic measure of unlearning cost which accounts for varying inference costs.
arXiv Detail & Related papers (2024-02-01T16:43:04Z) - Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning
Interference with Gradient Projection [56.292071534857946]
Recent data-privacy laws have sparked interest in machine unlearning.
Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset.
We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU)
We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
arXiv Detail & Related papers (2023-12-07T07:17:24Z) - Fast Machine Unlearning Without Retraining Through Selective Synaptic
Dampening [51.34904967046097]
Selective Synaptic Dampening (SSD) is a fast, performant, and does not require long-term storage of the training data.
We present a novel two-step, post hoc, retrain-free approach to machine unlearning which is fast, performant, and does not require long-term storage of the training data.
arXiv Detail & Related papers (2023-08-15T11:30:45Z) - AI Model Disgorgement: Methods and Choices [127.54319351058167]
We introduce a taxonomy of possible disgorgement methods that are applicable to modern machine learning systems.
We investigate the meaning of "removing the effects" of data in the trained model in a way that does not require retraining from scratch.
arXiv Detail & Related papers (2023-04-07T08:50:18Z) - Zero-Shot Machine Unlearning [6.884272840652062]
Modern privacy regulations grant citizens the right to be forgotten by products, services and companies.
No data related to the training process or training samples may be accessible for the unlearning purpose.
We propose two novel solutions for zero-shot machine unlearning based on (a) error minimizing-maximizing noise and (b) gated knowledge transfer.
arXiv Detail & Related papers (2022-01-14T19:16:09Z) - Machine Unlearning of Features and Labels [72.81914952849334]
We propose first scenarios for unlearning and labels in machine learning models.
Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
arXiv Detail & Related papers (2021-08-26T04:42:24Z) - SSSE: Efficiently Erasing Samples from Trained Machine Learning Models [103.43466657962242]
We propose an efficient and effective algorithm, SSSE, for samples erasure.
In certain cases SSSE can erase samples almost as well as the optimal, yet impractical, gold standard of training a new model from scratch with only the permitted data.
arXiv Detail & Related papers (2021-07-08T14:17:24Z) - Decentralized Federated Learning Preserves Model and Data Privacy [77.454688257702]
We propose a fully decentralized approach, which allows to share knowledge between trained models.
Students are trained on the output of their teachers via synthetically generated input data.
The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher.
arXiv Detail & Related papers (2021-02-01T14:38:54Z) - An Efficient Method of Training Small Models for Regression Problems
with Knowledge Distillation [1.433758865948252]
We propose a new formalism of knowledge distillation for regression problems.
First, we propose a new loss function, teacher outlier loss rejection, which rejects outliers in training samples using teacher model predictions.
By considering the multi-task network, training of the feature extraction of student models becomes more effective.
arXiv Detail & Related papers (2020-02-28T08:46:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.