Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2206.00518v2
• Date: Fri, 3 Jun 2022 00:25:48 GMT
• Title: Efficient Scheduling of Data Augmentation for Deep Reinforcement Learning
• Abstract summary: In deep reinforcement learning (RL), data augmentation is widely considered as a tool to induce a set of useful priors. But even when the prior is useful for generalization, distilling it to RL agent often interferes with RL training and degenerates sample efficiency. We devise a stand-alone network distillation method to inject the consistency prior at any time (even after RL) and a simple yet efficient framework to automatically schedule the distillation.
• Score: 2.837718256830632
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In deep reinforcement learning (RL), data augmentation is widely considered as a tool to induce a set of useful priors about semantic consistency and improve sample efficiency and generalization performance. However, even when the prior is useful for generalization, distilling it to RL agent often interferes with RL training and degenerates sample efficiency. Meanwhile, the agent is forgetful of the prior due to the non-stationary nature of RL. These observations suggest two extreme schedules of distillation: (i) over the entire training; or (ii) only at the end. Hence, we devise a stand-alone network distillation method to inject the consistency prior at any time (even after RL), and a simple yet efficient framework to automatically schedule the distillation. Specifically, the proposed framework first focuses on mastering train environments regardless of generalization by adaptively deciding which {\it or no} augmentation to be used for the training. After this, we add the distillation to extract the remaining benefits for generalization from all the augmentations, which requires no additional new samples. In our experiments, we demonstrate the utility of the proposed framework, in particular, that considers postponing the augmentation to the end of RL training.

Related papers

• Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversion [84.20493238687187]
  We introduce a novel framework that moves beyond simple mimicry to instill a deeper conceptual understanding.<n>underlinetextitFirst, to address pattern memorization, Explanatory Inversion (EI) generates targeted explanatory probes''<n>underlinetextitSecond, to improve generalization, Explanatory GRPO (texttEXGRPO) uses a reinforcement learning algorithm with a novel Dialogue Structure Utility Bonus.
  arXiv  Detail & Related papers  (2026-02-26T23:01:46Z)
• Fast and Effective On-policy Distillation from Reasoning Prefixes [5.0200371345178]
  On-policy distillation (OPD) samples trajectories from the student model and supervises them with a teacher at the token level.<n>We propose a simple yet effective modification of OPD: we apply the distillation objective only to prefixes of student-generated outputs and terminate each sampling early during distillation.<n>Experiments on a suite of AI-for-Math and out-of-domain benchmarks show that on-policy prefix distillation matches the performance of full OPD while reducing training FLOP by 2x-47x.
  arXiv  Detail & Related papers  (2026-02-16T23:28:54Z)
• Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Models [130.8912476550625]
  We propose a positive-unlabeled (PU) reinforcement learning distillation method for on-premise small-model deployment.<n>Our method distills the teacher's preference-optimization capability from black-box generations into a locally trainable student.<n>Experiments demonstrate that our method achieves consistently strong performance under a low-cost setting.
  arXiv  Detail & Related papers  (2026-01-28T15:14:50Z)
• Light distillation for Incremental Graph Convolution Collaborative Filtering [0.0]
  We propose a light preference-driven distillation method to distill the preference score of a user for an item directly from historical interactions.<n>The proposed method can save training time from 1.5x to 9.5x compared to the existing methods and improves Recall@20 by 5.41% and 10.64% from the fine-tune method.
  arXiv  Detail & Related papers  (2025-05-26T10:47:26Z)
• AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning [50.02117478165099]
  We show that large-scale reinforcement learning can significantly enhance the reasoning capabilities of strong, small- and mid-sized models.<n>We propose a simple yet effective approach: first training on math-only prompts, then on code-only prompts.
  arXiv  Detail & Related papers  (2025-05-22T08:50:47Z)
• A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learning [12.889687274108248]
  A Q-learning algorithm is prone to overfitting and training instabilities when trained from visual observations. We propose a generalized recipe, SADA, that works with wider varieties of augmentations. We find that our method, SADA, greatly improves training stability and generalization of RL agents across a diverse set of augmentations.
  arXiv  Detail & Related papers  (2024-05-27T17:58:23Z)
• RICE: Breaking Through the Training Bottlenecks of Reinforcement Learning with Explanation [40.84214941048131]
  RICE is an innovative refining scheme for reinforcement learning. It incorporates explanation methods to break through the training bottlenecks. We evaluate RICE in various popular RL environments and real-world applications.
  arXiv  Detail & Related papers  (2024-05-05T22:06:42Z)
• Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality [78.6359306550245]
  We argue that using just one synthetic subset for distillation will not yield optimal generalization performance. PDD synthesizes multiple small sets of synthetic images, each conditioned on the previous sets, and trains the model on the cumulative union of these subsets. Our experiments show that PDD can effectively improve the performance of existing dataset distillation methods by up to 4.3%.
  arXiv  Detail & Related papers  (2023-10-10T20:04:44Z)
• Distill Gold from Massive Ores: Bi-level Data Pruning towards Efficient Dataset Distillation [96.92250565207017]
  We study the data efficiency and selection for the dataset distillation task. By re-formulating the dynamics of distillation, we provide insight into the inherent redundancy in the real dataset. We find the most contributing samples based on their causal effects on the distillation.
  arXiv  Detail & Related papers  (2023-05-28T06:53:41Z)
• Single-Shot Pruning for Offline Reinforcement Learning [47.886329599997474]
  Deep Reinforcement Learning (RL) is a powerful framework for solving complex real-world problems. One way to tackle this problem is to prune neural networks leaving only the necessary parameters. We close the gap between RL and single-shot pruning techniques and present a general pruning approach to the Offline RL.
  arXiv  Detail & Related papers  (2021-12-31T18:10:02Z)
• Stochastic Training is Not Necessary for Generalization [57.04880404584737]
  It is widely believed that the implicit regularization of gradient descent (SGD) is fundamental to the impressive generalization behavior we observe in neural networks. In this work, we demonstrate that non-stochastic full-batch training can achieve strong performance on CIFAR-10 that is on-par with SGD.
  arXiv  Detail & Related papers  (2021-09-29T00:50:00Z)
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation [0.0]
  Self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. We consider an iterative variant of self-distillation in a kernel regression setting, in which successive steps incorporate both model outputs and the ground-truth targets. We show that any such function obtained with self-distillation can be calculated directly as a function of the initial fit, and that infinite distillation steps yields the same optimization problem as the original with amplified regularization.
  arXiv  Detail & Related papers  (2021-02-25T18:56:09Z)
• Time Matters in Using Data Augmentation for Vision-based Deep Reinforcement Learning [4.921588282642753]
  The timing of using augmentation, which is, in turn, critical depending on tasks to be solved in training and testing. If the regularization imposed by augmentation is helpful only in testing, it is better to procrastinate the augmentation after training than to use it during training in terms of sample and computation complexity.
  arXiv  Detail & Related papers  (2021-02-17T05:22:34Z)
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning [90.93035276307239]
  We propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents. We demonstrate the extreme generalization benefits of our approach in different domains ranging from maze navigation to robotic tasks. This work provides a principled way to improve generalization in RL by gradually removing information that is redundant for task-solving.
  arXiv  Detail & Related papers  (2020-08-03T02:24:20Z)
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning [67.34810824996887]
  Non-stationarity can arise in Reinforcement Learning (RL) even in stationary environments. We propose Iterated Relearning (ITER) to improve generalisation of deep RL agents.
  arXiv  Detail & Related papers  (2020-06-10T13:26:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.