Ranking Neural Checkpoints

• URL: http://arxiv.org/abs/2011.11200v4
• Date: Sun, 28 Aug 2022 03:17:13 GMT
• Title: Ranking Neural Checkpoints
• Authors: Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green, Liqiang Wang, Boqing Gong
• Abstract summary: This paper is concerned with ranking pre-trained deep neural networks (DNNs) for the transfer learning to a downstream task. We establish a neural checkpoint ranking benchmark (NeuCRaB) and study some intuitive ranking measures. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability.
• Score: 57.27352551718646
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper is concerned with ranking many pre-trained deep neural networks (DNNs), called checkpoints, for the transfer learning to a downstream task. Thanks to the broad use of DNNs, we may easily collect hundreds of checkpoints from various sources. Which of them transfers the best to our downstream task of interest? Striving to answer this question thoroughly, we establish a neural checkpoint ranking benchmark (NeuCRaB) and study some intuitive ranking measures. These measures are generic, applying to the checkpoints of different output types without knowing how the checkpoints are pre-trained on which dataset. They also incur low computation cost, making them practically meaningful. Our results suggest that the linear separability of the features extracted by the checkpoints is a strong indicator of transferability. We also arrive at a new ranking measure, NLEEP, which gives rise to the best performance in the experiments.

Related papers

• Unveiling the Power of Sparse Neural Networks for Feature Selection [60.50319755984697]
  Sparse Neural Networks (SNNs) have emerged as powerful tools for efficient feature selection. We show that SNNs trained with dynamic sparse training (DST) algorithms can achieve, on average, more than $50%$ memory and $55%$ FLOPs reduction. Our findings show that feature selection with SNNs trained with DST algorithms can achieve, on average, more than $50%$ memory and $55%$ FLOPs reduction.
  arXiv  Detail & Related papers  (2024-08-08T16:48:33Z)
• SAfER: Layer-Level Sensitivity Assessment for Efficient and Robust Neural Network Inference [20.564198591600647]
  Deep neural networks (DNNs) demonstrate outstanding performance across most computer vision tasks. Some critical applications, such as autonomous driving or medical imaging, also require investigation into their behavior. DNN attribution consists in studying the relationship between the predictions of a DNN and its inputs.
  arXiv  Detail & Related papers  (2023-08-09T07:45:51Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• RankDNN: Learning to Rank for Few-shot Learning [70.49494297554537]
  This paper introduces a new few-shot learning pipeline that casts relevance ranking for image retrieval as binary ranking relation classification. It provides a new perspective on few-shot learning and is complementary to state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-11-28T13:59:31Z)
• Revisiting Checkpoint Averaging for Neural Machine Translation [44.37101354412253]
  Checkpoint averaging is a simple and effective method to boost the performance of converged neural machine translation models. In this work, we revisit the concept of checkpoint averaging and consider several extensions.
  arXiv  Detail & Related papers  (2022-10-21T08:29:23Z)
• Continual Prune-and-Select: Class-incremental learning with specialized subnetworks [66.4795381419701]
  Continual-Prune-and-Select (CP&S) is capable of sequentially learning 10 tasks from ImageNet-1000 keeping an accuracy around 94% with negligible forgetting. This is a first-of-its-kind result in class-incremental learning.
  arXiv  Detail & Related papers  (2022-08-09T10:49:40Z)
• CheckSel: Efficient and Accurate Data-valuation Through Online Checkpoint Selection [3.321404824316694]
  We propose a novel 2-phase solution to the problem of data valuation and subset selection. Phase 1 selects representative checkpoints from an SGD-like training algorithm, which are used in phase-2 to estimate the approximate training data values. Experimental results show the proposed algorithm outperforms recent baseline methods by up to 30% in terms of test accuracy.
  arXiv  Detail & Related papers  (2022-03-14T02:06:52Z)
• Ranking and Rejecting of Pre-Trained Deep Neural Networks in Transfer Learning based on Separation Index [0.16058099298620418]
  We introduce an algorithm to rank pre-trained Deep Neural Networks (DNNs) by applying a distance-based complexity measure named Separation Index (SI) to the target dataset. The efficiency of the proposed algorithm is evaluated by using three challenging datasets including Linnaeus 5, Breast Cancer Images, and COVID-CT.
  arXiv  Detail & Related papers  (2020-12-26T11:14:12Z)
• Sequential Changepoint Detection in Neural Networks with Checkpoints [11.763229353978321]
  We introduce a framework for online changepoint detection and simultaneous model learning. It is based on detecting changepoints across time by sequentially performing generalized likelihood ratio tests. We show improved performance compared to online Bayesian changepoint detection.
  arXiv  Detail & Related papers  (2020-10-06T21:49:54Z)
• COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization [60.07531696857743]
  Co-Learns DNNs and soft labels through Alternating Minimization of two objectives. We propose COLAM framework that Co-Learns DNNs and soft labels through Alternating Minimization of two objectives.
  arXiv  Detail & Related papers  (2020-04-26T17:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.