Importance Sampling for Stochastic Gradient Descent in Deep Neural Networks

• URL: http://arxiv.org/abs/2303.16529v1
• Date: Wed, 29 Mar 2023 08:35:11 GMT
• Title: Importance Sampling for Stochastic Gradient Descent in Deep Neural Networks
• Authors: Thibault Lahire
• Abstract summary: Importance sampling for training deep neural networks has been widely studied. This paper reviews the challenges inherent to this research area. We propose a metric allowing the assessment of the quality of a given sampling scheme.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Stochastic gradient descent samples uniformly the training set to build an unbiased gradient estimate with a limited number of samples. However, at a given step of the training process, some data are more helpful than others to continue learning. Importance sampling for training deep neural networks has been widely studied to propose sampling schemes yielding better performance than the uniform sampling scheme. After recalling the theory of importance sampling for deep learning, this paper reviews the challenges inherent to this research area. In particular, we propose a metric allowing the assessment of the quality of a given sampling scheme; and we study the interplay between the sampling scheme and the optimizer used.

Related papers

• Adaptive teachers for amortized samplers [76.88721198565861]
  Amortized inference is the task of training a parametric model, such as a neural network, to approximate a distribution with a given unnormalized density where exact sampling is intractable. Off-policy RL training facilitates the discovery of diverse, high-reward candidates, but existing methods still face challenges in efficient exploration. We propose an adaptive training distribution (the Teacher) to guide the training of the primary amortized sampler (the Student) by prioritizing high-loss regions.
  arXiv  Detail & Related papers  (2024-10-02T11:33:13Z)
• Data Pruning via Moving-one-Sample-out [61.45441981346064]
  We propose a novel data-pruning approach called moving-one-sample-out (MoSo) MoSo aims to identify and remove the least informative samples from the training set. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios.
  arXiv  Detail & Related papers  (2023-10-23T08:00:03Z)
• Sampling Through the Lens of Sequential Decision Making [9.101505546901999]
  We propose a reward-guided sampling strategy called Adaptive Sample with Reward (ASR) Our approach optimally adjusts the sampling process to achieve optimal performance. Empirical results in information retrieval and clustering demonstrate ASR's superb performance across different datasets.
  arXiv  Detail & Related papers  (2022-08-17T04:01:29Z)
• Active Learning for Deep Visual Tracking [51.5063680734122]
  Convolutional neural networks (CNNs) have been successfully applied to the single target tracking task in recent years. In this paper, we propose an active learning method for deep visual tracking, which selects and annotates the unlabeled samples to train the deep CNNs model. Under the guidance of active learning, the tracker based on the trained deep CNNs model can achieve competitive tracking performance while reducing the labeling cost.
  arXiv  Detail & Related papers  (2021-10-17T11:47:56Z)
• Beyond Farthest Point Sampling in Point-Wise Analysis [52.218037492342546]
  We propose a novel data-driven sampler learning strategy for point-wise analysis tasks. We learn sampling and downstream applications jointly. Our experiments show that jointly learning of the sampler and task brings remarkable improvement over previous baseline methods.
  arXiv  Detail & Related papers  (2021-07-09T08:08:44Z)
• AutoSampling: Search for Effective Data Sampling Schedules [118.20014773014671]
  We propose an AutoSampling method to automatically learn sampling schedules for model training. We apply our method to a variety of image classification tasks illustrating the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2021-05-28T09:39:41Z)
• Robust Ensemble Model Training via Random Layer Sampling Against Adversarial Attack [38.1887818626171]
  We propose an ensemble model training framework with random layer sampling to improve the robustness of deep neural networks. In the proposed training framework, we generate various sampled model through the random layer sampling and update the weight of the sampled model. After the ensemble models are trained, it can hide the gradient efficiently and avoid the gradient-based attack.
  arXiv  Detail & Related papers  (2020-05-21T16:14:18Z)
• Efficient Deep Representation Learning by Adaptive Latent Space Sampling [16.320898678521843]
  Supervised deep learning requires a large amount of training samples with annotations, which are expensive and time-consuming to obtain. We propose a novel training framework which adaptively selects informative samples that are fed to the training process.
  arXiv  Detail & Related papers  (2020-03-19T22:17:02Z)
• Subset Sampling For Progressive Neural Network Learning [106.12874293597754]
  Progressive Neural Network Learning is a class of algorithms that incrementally construct the network's topology and optimize its parameters based on the training data. We propose to speed up this process by exploiting subsets of training data at each incremental training step. Experimental results in object, scene and face recognition problems demonstrate that the proposed approach speeds up the optimization procedure considerably.
  arXiv  Detail & Related papers  (2020-02-17T18:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.