Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models
- URL: http://arxiv.org/abs/2312.14923v1
- Date: Fri, 22 Dec 2023 18:55:45 GMT
- Title: Fast-NTK: Parameter-Efficient Unlearning for Large-Scale Models
- Authors: Guihong Li, Hsiang Hsu, Chun-Fu Chen, and Radu Marculescu
- Abstract summary: machine unlearning'' proposes the selective removal of unwanted data without the need for retraining from scratch.
Fast-NTK is a novel NTK-based unlearning algorithm that significantly reduces the computational complexity.
- Score: 17.34908967455907
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The rapid growth of machine learning has spurred legislative initiatives such
as ``the Right to be Forgotten,'' allowing users to request data removal. In
response, ``machine unlearning'' proposes the selective removal of unwanted
data without the need for retraining from scratch. While the
Neural-Tangent-Kernel-based (NTK-based) unlearning method excels in
performance, it suffers from significant computational complexity, especially
for large-scale models and datasets. Our work introduces ``Fast-NTK,'' a novel
NTK-based unlearning algorithm that significantly reduces the computational
complexity by incorporating parameter-efficient fine-tuning methods, such as
fine-tuning batch normalization layers in a CNN or visual prompts in a vision
transformer. Our experimental results demonstrate scalability to much larger
neural networks and datasets (e.g., 88M parameters; 5k images), surpassing the
limitations of previous full-model NTK-based approaches designed for smaller
cases (e.g., 8M parameters; 500 images). Notably, our approach maintains a
performance comparable to the traditional method of retraining on the retain
set alone. Fast-NTK can thus enable for practical and scalable NTK-based
unlearning in deep neural networks.
Related papers
- Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch [72.26822499434446]
Auto-Train-Once (ATO) is an innovative network pruning algorithm designed to automatically reduce the computational and storage costs of DNNs.
We provide a comprehensive convergence analysis as well as extensive experiments, and the results show that our approach achieves state-of-the-art performance across various model architectures.
arXiv Detail & Related papers (2024-03-21T02:33:37Z) - Solving Large-scale Spatial Problems with Convolutional Neural Networks [88.31876586547848]
We employ transfer learning to improve training efficiency for large-scale spatial problems.
We propose that a convolutional neural network (CNN) can be trained on small windows of signals, but evaluated on arbitrarily large signals with little to no performance degradation.
arXiv Detail & Related papers (2023-06-14T01:24:42Z) - Emulation Learning for Neuromimetic Systems [0.0]
Building on our recent research on neural quantization systems, results on learning quantized motions and resilience to channel dropouts are reported.
We propose a general Deep Q Network (DQN) algorithm that can not only learn the trajectory but also exhibit the advantages of resilience to channel dropout.
arXiv Detail & Related papers (2023-05-04T22:47:39Z) - Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency.
We present an IPU-optimized release of our custom SNN Python package, snnTorch.
arXiv Detail & Related papers (2022-11-19T15:44:08Z) - Precision Machine Learning [5.15188009671301]
We compare various function approximation methods and study how they scale with increasing parameters and data.
We find that neural networks can often outperform classical approximation methods on high-dimensional examples.
We develop training tricks which enable us to train neural networks to extremely low loss, close to the limits allowed by numerical precision.
arXiv Detail & Related papers (2022-10-24T17:58:30Z) - Go Beyond Multiple Instance Neural Networks: Deep-learning Models based
on Local Pattern Aggregation [0.0]
convolutional neural networks (CNNs) have brought breakthroughs in processing clinical electrocardiograms (ECGs) and speaker-independent speech.
In this paper, we propose local pattern aggregation-based deep-learning models to effectively deal with both problems.
The novel network structure, called LPANet, has cropping and aggregation operations embedded into it.
arXiv Detail & Related papers (2022-05-28T13:18:18Z) - Dynamic Neural Diversification: Path to Computationally Sustainable
Neural Networks [68.8204255655161]
Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks.
We explore the diversity of the neurons within the hidden layer during the learning process.
We analyze how the diversity of the neurons affects predictions of the model.
arXiv Detail & Related papers (2021-09-20T15:12:16Z) - Training Deep Neural Networks with Constrained Learning Parameters [4.917317902787792]
A significant portion of deep learning tasks would run on edge computing systems.
We propose the Combinatorial Neural Network Training Algorithm (CoNNTrA)
CoNNTrA trains deep learning models with ternary learning parameters on the MNIST, Iris and ImageNet data sets.
Our results indicate that CoNNTrA models use 32x less memory and have errors at par with the Backpropagation models.
arXiv Detail & Related papers (2020-09-01T16:20:11Z) - Progressive Tandem Learning for Pattern Recognition with Deep Spiking
Neural Networks [80.15411508088522]
Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency.
We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
arXiv Detail & Related papers (2020-07-02T15:38:44Z) - Large-Scale Gradient-Free Deep Learning with Recursive Local
Representation Alignment [84.57874289554839]
Training deep neural networks on large-scale datasets requires significant hardware resources.
Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize.
We propose a neuro-biologically-plausible alternative to backprop that can be used to train deep networks.
arXiv Detail & Related papers (2020-02-10T16:20:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.