COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization

• URL: http://arxiv.org/abs/2004.12443v1
• Date: Sun, 26 Apr 2020 17:50:20 GMT
• Title: COLAM: Co-Learning of Deep Neural Networks and Soft Labels via Alternating Minimization
• Authors: Xingjian Li, Haoyi Xiong, Haozhe An, Dejing Dou, Chengzhong Xu
• Abstract summary: Co-Learns DNNs and soft labels through Alternating Minimization of two objectives. We propose COLAM framework that Co-Learns DNNs and soft labels through Alternating Minimization of two objectives.
• Score: 60.07531696857743
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Softening labels of training datasets with respect to data representations has been frequently used to improve the training of deep neural networks (DNNs). While such a practice has been studied as a way to leverage privileged information about the distribution of the data, a well-trained learner with soft classification outputs should be first obtained as a prior to generate such privileged information. To solve such chicken-egg problem, we propose COLAM framework that Co-Learns DNNs and soft labels through Alternating Minimization of two objectives - (a) the training loss subject to soft labels and (b) the objective to learn improved soft labels - in one end-to-end training procedure. We performed extensive experiments to compare our proposed method with a series of baselines. The experiment results show that COLAM achieves improved performance on many tasks with better testing classification accuracy. We also provide both qualitative and quantitative analyses that explain why COLAM works well.

Related papers

• KAKURENBO: Adaptively Hiding Samples in Deep Neural Network Training [2.8804804517897935]
  We propose a method for hiding the least-important samples during the training of deep neural networks. We adaptively find samples to exclude in a given epoch based on their contribution to the overall learning process. Our method can reduce total training time by up to 22% impacting accuracy only by 0.4% compared to the baseline.
  arXiv  Detail & Related papers  (2023-10-16T06:19:29Z)
• SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learning [101.86916775218403]
  This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation. We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training. In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
  arXiv  Detail & Related papers  (2023-01-26T03:53:25Z)
• Synergistic Network Learning and Label Correction for Noise-robust Image Classification [28.27739181560233]
  Deep Neural Networks (DNNs) tend to overfit training label noise, resulting in poorer model performance in practice. We propose a robust label correction framework combining the ideas of small loss selection and noise correction. We demonstrate our method on both synthetic and real-world datasets with different noise types and rates.
  arXiv  Detail & Related papers  (2022-02-27T23:06:31Z)
• Active Learning for Deep Visual Tracking [51.5063680734122]
  Convolutional neural networks (CNNs) have been successfully applied to the single target tracking task in recent years. In this paper, we propose an active learning method for deep visual tracking, which selects and annotates the unlabeled samples to train the deep CNNs model. Under the guidance of active learning, the tracker based on the trained deep CNNs model can achieve competitive tracking performance while reducing the labeling cost.
  arXiv  Detail & Related papers  (2021-10-17T11:47:56Z)
• When Deep Learners Change Their Mind: Learning Dynamics for Active Learning [32.792098711779424]
  In this paper, we propose a new informativeness-based active learning method. Our measure is derived from the learning dynamics of a neural network. We show that label-dispersion is a promising predictor of the uncertainty of the network.
  arXiv  Detail & Related papers  (2021-07-30T15:30:17Z)
• Learning Robust Variational Information Bottleneck with Reference [12.743882133781598]
  We propose a new approach to train a variational information bottleneck (VIB) that improves its robustness to adversarial perturbations. We refine the categorical class information in the training phase with soft labels which are obtained from a pre-trained reference neural network.
  arXiv  Detail & Related papers  (2021-04-29T14:46:09Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• Delving Deep into Label Smoothing [112.24527926373084]
  Label smoothing is an effective regularization tool for deep neural networks (DNNs) We present an Online Label Smoothing (OLS) strategy, which generates soft labels based on the statistics of the model prediction for the target category.
  arXiv  Detail & Related papers  (2020-11-25T08:03:11Z)
• Temporal Calibrated Regularization for Robust Noisy Label Learning [60.90967240168525]
  Deep neural networks (DNNs) exhibit great success on many tasks with the help of large-scale well annotated datasets. However, labeling large-scale data can be very costly and error-prone so that it is difficult to guarantee the annotation quality. We propose a Temporal Calibrated Regularization (TCR) in which we utilize the original labels and the predictions in the previous epoch together.
  arXiv  Detail & Related papers  (2020-07-01T04:48:49Z)
• Iterative Label Improvement: Robust Training by Confidence Based Filtering and Dataset Partitioning [5.1293809610257775]
  State-of-the-art, high capacity deep neural networks require large amounts of labelled training data. They are also highly susceptible to label errors in this data. We propose a novel meta training and labelling scheme that is able to use inexpensive unlabelled data.
  arXiv  Detail & Related papers  (2020-02-07T10:42:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.