Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced
Transfer Learning
- URL: http://arxiv.org/abs/2310.08782v3
- Date: Sat, 18 Nov 2023 16:08:35 GMT
- Title: Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced
Transfer Learning
- Authors: Yihua Zhang, Yimeng Zhang, Aochuan Chen, Jinghan Jia, Jiancheng Liu,
Gaowen Liu, Mingyi Hong, Shiyu Chang, Sijia Liu
- Abstract summary: dataset pruning (DP) has emerged as an effective way to improve data efficiency.
We propose two new DP methods, label mapping and feature mapping, for supervised and self-supervised pretraining settings.
We show that source data classes can be pruned by up to 40% 80% without sacrificing downstream performance.
- Score: 66.20311762506702
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Massive data is often considered essential for deep learning applications,
but it also incurs significant computational and infrastructural costs.
Therefore, dataset pruning (DP) has emerged as an effective way to improve data
efficiency by identifying and removing redundant training samples without
sacrificing performance. In this work, we aim to address the problem of DP for
transfer learning, i.e., how to prune a source dataset for improved pretraining
efficiency and lossless finetuning accuracy on downstream target tasks. To our
best knowledge, the problem of DP for transfer learning remains open, as
previous studies have primarily addressed DP and transfer learning as separate
problems. By contrast, we establish a unified viewpoint to integrate DP with
transfer learning and find that existing DP methods are not suitable for the
transfer learning paradigm. We then propose two new DP methods, label mapping
and feature mapping, for supervised and self-supervised pretraining settings
respectively, by revisiting the DP problem through the lens of source-target
domain mapping. Furthermore, we demonstrate the effectiveness of our approach
on numerous transfer learning tasks. We show that source data classes can be
pruned by up to 40% ~ 80% without sacrificing downstream performance, resulting
in a significant 2 ~ 5 times speed-up during the pretraining stage. Besides,
our proposal exhibits broad applicability and can improve other computationally
intensive transfer learning techniques, such as adversarial pretraining. Codes
are available at https://github.com/OPTML-Group/DP4TL.
Related papers
- DPFormer: Learning Differentially Private Transformer on Long-Tailed
Data [6.848321493051996]
The Transformer has emerged as a versatile and effective architecture with broad applications.
It still remains an open problem how to efficiently train a Transformer model of high utility with differential privacy guarantees.
In this paper, we identify two key challenges in learning differentially private Transformers, i.e., heavy computation overhead due to per-sample gradient clipping and unintentional attention distraction within the attention mechanism.
We propose DPFormer, equipped with Phantom Clipping and Re-Attention Mechanism, to address these challenges.
arXiv Detail & Related papers (2023-05-28T05:00:07Z) - Optimal transfer protocol by incremental layer defrosting [66.76153955485584]
Transfer learning is a powerful tool enabling model training with limited amounts of data.
The simplest transfer learning protocol is based on freezing" the feature-extractor layers of a network pre-trained on a data-rich source task.
We show that this protocol is often sub-optimal and the largest performance gain may be achieved when smaller portions of the pre-trained network are kept frozen.
arXiv Detail & Related papers (2023-03-02T17:32:11Z) - Frozen Overparameterization: A Double Descent Perspective on Transfer
Learning of Deep Neural Networks [27.17697714584768]
We study the generalization behavior of transfer learning of deep neural networks (DNNs)
We show that the test error evolution during the target training has a more significant double descent effect when the target training dataset is sufficiently large.
Also, we show that the double descent phenomenon may make a transfer from a less related source task better than a transfer from a more related source task.
arXiv Detail & Related papers (2022-11-20T20:26:23Z) - Segmentation-guided Domain Adaptation for Efficient Depth Completion [3.441021278275805]
We propose an efficient depth completion model based on a vgg05-like CNN architecture and a semi-supervised domain adaptation approach.
In order to boost spatial coherence, we guide the learning process using segmentations as additional source of information.
Our approach improves on previous efficient and low parameter state of the art approaches while having a noticeably lower computational footprint.
arXiv Detail & Related papers (2022-10-14T13:01:25Z) - Knowledge Distillation as Efficient Pre-training: Faster Convergence,
Higher Data-efficiency, and Better Transferability [53.27240222619834]
Knowledge Distillation as Efficient Pre-training aims to efficiently transfer the learned feature representation from pre-trained models to new student models for future downstream tasks.
Our method performs comparably with supervised pre-training counterparts in 3 downstream tasks and 9 downstream datasets requiring 10x less data and 5x less pre-training time.
arXiv Detail & Related papers (2022-03-10T06:23:41Z) - Invariance Learning in Deep Neural Networks with Differentiable Laplace
Approximations [76.82124752950148]
We develop a convenient gradient-based method for selecting the data augmentation.
We use a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective.
arXiv Detail & Related papers (2022-02-22T02:51:11Z) - Meta-learning Transferable Representations with a Single Target Domain [46.83481356352768]
Fine-tuning and joint training do not always improve accuracy on downstream tasks.
We propose Meta Representation Learning (MeRLin) to learn transferable features.
MeRLin empirically outperforms previous state-of-the-art transfer learning algorithms on various real-world vision and NLP transfer learning benchmarks.
arXiv Detail & Related papers (2020-11-03T01:57:37Z) - Towards Accurate Knowledge Transfer via Target-awareness Representation
Disentanglement [56.40587594647692]
We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED)
TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model.
Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
arXiv Detail & Related papers (2020-10-16T17:45:08Z) - Parameter-Efficient Transfer from Sequential Behaviors for User Modeling
and Recommendation [111.44445634272235]
In this paper, we develop a parameter efficient transfer learning architecture, termed as PeterRec.
PeterRec allows the pre-trained parameters to remain unaltered during fine-tuning by injecting a series of re-learned neural networks.
We perform extensive experimental ablation to show the effectiveness of the learned user representation in five downstream tasks.
arXiv Detail & Related papers (2020-01-13T14:09:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.