Variational Bayesian Unlearning
- URL: http://arxiv.org/abs/2010.12883v1
- Date: Sat, 24 Oct 2020 11:53:00 GMT
- Title: Variational Bayesian Unlearning
- Authors: Quoc Phong Nguyen, Bryan Kian Hsiang Low, Patrick Jaillet
- Abstract summary: We study the problem of approximately unlearning a Bayesian model from a small subset of the training data to be erased.
We show that it is equivalent to minimizing an evidence upper bound which trades off between fully unlearning from erased data vs. not entirely forgetting the posterior belief.
In model training with VI, only an approximate (instead of exact) posterior belief given the full data can be obtained, which makes unlearning even more challenging.
- Score: 54.26984662139516
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper studies the problem of approximately unlearning a Bayesian model
from a small subset of the training data to be erased. We frame this problem as
one of minimizing the Kullback-Leibler divergence between the approximate
posterior belief of model parameters after directly unlearning from erased data
vs. the exact posterior belief from retraining with remaining data. Using the
variational inference (VI) framework, we show that it is equivalent to
minimizing an evidence upper bound which trades off between fully unlearning
from erased data vs. not entirely forgetting the posterior belief given the
full data (i.e., including the remaining data); the latter prevents
catastrophic unlearning that can render the model useless. In model training
with VI, only an approximate (instead of exact) posterior belief given the full
data can be obtained, which makes unlearning even more challenging. We propose
two novel tricks to tackle this challenge. We empirically demonstrate our
unlearning methods on Bayesian models such as sparse Gaussian process and
logistic regression using synthetic and real-world datasets.
Related papers
- Why Fine-Tuning Struggles with Forgetting in Machine Unlearning? Theoretical Insights and a Remedial Approach [19.307968983872588]
Fine-tuning (FT) methods have become one of the fundamental approaches for approximating unlearning.
We present the first theoretical analysis of FT methods for machine unlearning within a linear regression framework.
We propose a theoretical approach to mitigate the retention of forgetting data in the pretrained model.
arXiv Detail & Related papers (2024-10-04T18:01:52Z) - Generalized Regression with Conditional GANs [2.4171019220503402]
We propose to learn a prediction function whose outputs, when paired with the corresponding inputs, are indistinguishable from feature-label pairs in the training dataset.
We show that this approach to regression makes fewer assumptions on the distribution of the data we are fitting to and, therefore, has better representation capabilities.
arXiv Detail & Related papers (2024-04-21T01:27:47Z) - Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data [49.73114504515852]
We show that replacing the original real data by each generation's synthetic data does indeed tend towards model collapse.
We demonstrate that accumulating the successive generations of synthetic data alongside the original real data avoids model collapse.
arXiv Detail & Related papers (2024-04-01T18:31:24Z) - Enhancing Consistency and Mitigating Bias: A Data Replay Approach for
Incremental Learning [100.7407460674153]
Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks.
To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks.
However, it is not expected in practice considering the memory constraint or data privacy issue.
As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
arXiv Detail & Related papers (2024-01-12T12:51:12Z) - Learning Defect Prediction from Unrealistic Data [57.53586547895278]
Pretrained models of code have become popular choices for code understanding and generation tasks.
Such models tend to be large and require commensurate volumes of training data.
It has become popular to train models with far larger but less realistic datasets, such as functions with artificially injected bugs.
Models trained on such data tend to only perform well on similar data, while underperforming on real world programs.
arXiv Detail & Related papers (2023-11-02T01:51:43Z) - Analysis of Interpolating Regression Models and the Double Descent
Phenomenon [3.883460584034765]
It is commonly assumed that models which interpolate noisy training data are poor to generalize.
The best models obtained are overparametrized and the testing error exhibits the double descent behavior as the model order increases.
We derive a result based on the behavior of the smallest singular value of the regression matrix that explains the peak location and the double descent shape of the testing error as a function of model order.
arXiv Detail & Related papers (2023-04-17T09:44:33Z) - Deep Regression Unlearning [6.884272840652062]
We introduce deep regression unlearning methods that generalize well and are robust to privacy attacks.
We conduct regression unlearning experiments for computer vision, natural language processing and forecasting applications.
arXiv Detail & Related papers (2022-10-15T05:00:20Z) - Learning from aggregated data with a maximum entropy model [73.63512438583375]
We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis.
We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
arXiv Detail & Related papers (2022-10-05T09:17:27Z) - Challenges and Pitfalls of Bayesian Unlearning [6.931200003384123]
Machine unlearning refers to the task of removing a subset of training data, thereby removing its contributions to a trained model.
Approximate unlearning are one class of methods for this task which avoid the need to retrain the model from scratch on the retained data.
Bayes' rule can be used to cast approximate unlearning as an inference problem where the objective is to obtain the updated posterior by dividing out the likelihood of deleted data.
arXiv Detail & Related papers (2022-07-07T11:24:50Z) - Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective.
Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination.
Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
arXiv Detail & Related papers (2021-05-18T15:13:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.