Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported

• URL: http://arxiv.org/abs/2405.15583v1
• Date: Fri, 24 May 2024 14:12:23 GMT
• Title: Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported
• Authors: Ethan Harvey, Mikhail Petrov, Michael C. Hughes,
• Abstract summary: We compare transfer learning with and without source task informed priors across 5 datasets. For the scenario of 5-300 examples per class, we find negative or negligible gains on 2 datasets, modest gains on 2 other datasets, and substantial gains on one dataset.
• Score: 4.453137996095194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We pursue transfer learning to improve classifier accuracy on a target task with few labeled examples available for training. Recent work suggests that using a source task to learn a prior distribution over neural net weights, not just an initialization, can boost target task performance. In this study, we carefully compare transfer learning with and without source task informed priors across 5 datasets. We find that standard transfer learning informed by an initialization only performs far better than reported in previous comparisons. The relative gains of methods using informative priors over standard transfer learning vary in magnitude across datasets. For the scenario of 5-300 examples per class, we find negative or negligible gains on 2 datasets, modest gains (between 1.5-3 points of accuracy) on 2 other datasets, and substantial gains (>8 points) on one dataset. Among methods using informative priors, we find that an isotropic covariance appears competitive with learned low-rank covariance matrix while being substantially simpler to understand and tune. Further analysis suggests that the mechanistic justification for informed priors -- hypothesized improved alignment between train and test loss landscapes -- is not consistently supported due to high variability in empirical landscapes. We release code to allow independent reproduction of all experiments.

Related papers

• Data Adaptive Traceback for Vision-Language Foundation Models in Image Classification [34.37262622415682]
  We propose a new adaptation framework called Data Adaptive Traceback. Specifically, we utilize a zero-shot-based method to extract the most downstream task-related subset of the pre-training data. We adopt a pseudo-label-based semi-supervised technique to reuse the pre-training images and a vision-language contrastive learning method to address the confirmation bias issue in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-07-11T18:01:58Z)
• Diffusion-based Neural Network Weights Generation [85.6725307453325]
  We propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [100.7407460674153]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks. To mitigate the problem, a line of methods propose to replay the data of experienced tasks when learning new tasks. However, it is not expected in practice considering the memory constraint or data privacy issue. As a replacement, data-free data replay methods are proposed by inverting samples from the classification model.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Neural Priming for Sample-Efficient Adaptation [92.14357804106787]
  We propose Neural Priming, a technique for adapting large pretrained models to distribution shifts and downstream tasks. Neural Priming can be performed at test time, even for pretraining as large as LAION-2B.
  arXiv  Detail & Related papers  (2023-06-16T21:53:16Z)
• A Data-Based Perspective on Transfer Learning [76.30206800557411]
  We take a closer look at the role of the source dataset's composition in transfer learning. Our framework gives rise to new capabilities such as pinpointing transfer learning brittleness.
  arXiv  Detail & Related papers  (2022-07-12T17:58:28Z)
• Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors [59.93972277761501]
  We show that we can learn highly informative posteriors from the source task, through supervised or self-supervised approaches. This simple modular approach enables significant performance gains and more data-efficient learning on a variety of downstream classification and segmentation tasks.
  arXiv  Detail & Related papers  (2022-05-20T16:19:30Z)
• ProtoDA: Efficient Transfer Learning for Few-Shot Intent Classification [21.933876113300897]
  We adopt an alternative approach by transfer learning on an ensemble of related tasks using prototypical networks under the meta-learning paradigm. Using intent classification as a case study, we demonstrate that increasing variability in training tasks can significantly improve classification performance.
  arXiv  Detail & Related papers  (2021-01-28T00:19:13Z)
• Meta-learning Transferable Representations with a Single Target Domain [46.83481356352768]
  Fine-tuning and joint training do not always improve accuracy on downstream tasks. We propose Meta Representation Learning (MeRLin) to learn transferable features. MeRLin empirically outperforms previous state-of-the-art transfer learning algorithms on various real-world vision and NLP transfer learning benchmarks.
  arXiv  Detail & Related papers  (2020-11-03T01:57:37Z)
• Do Adversarially Robust ImageNet Models Transfer Better? [102.09335596483695]
  adversarially robust models often perform better than their standard-trained counterparts when used for transfer learning. Our results are consistent with (and in fact, add to) recent hypotheses stating that robustness leads to improved feature representations.
  arXiv  Detail & Related papers  (2020-07-16T17:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.