On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2105.01648v1
• Date: Tue, 4 May 2021 17:47:39 GMT
• Title: On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
• Authors: Marc Aurel Vischer, Robert Tjarko Lange, Henning Sprekeler
• Abstract summary: We show that feed-forward networks trained via supervised policy distillation and reinforcement learning can be pruned to the same level of sparsity. Using a set of carefully designed baseline conditions, we find that the majority of the lottery ticket effect in reinforcement learning can be attributed to the identified mask.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The lottery ticket hypothesis questions the role of overparameterization in supervised deep learning. But how does the distributional shift inherent to the reinforcement learning problem affect the performance of winning lottery tickets? In this work, we show that feed-forward networks trained via supervised policy distillation and reinforcement learning can be pruned to the same level of sparsity. Furthermore, we establish the existence of winning tickets for both on- and off-policy methods in a visual navigation and classic control task. Using a set of carefully designed baseline conditions, we find that the majority of the lottery ticket effect in reinforcement learning can be attributed to the identified mask. The resulting masked observation space eliminates redundant information and yields minimal task-relevant representations. The mask identified by iterative magnitude pruning provides an interpretable inductive bias. Its costly generation can be amortized by training dense agents with low-dimensional input and thereby at lower computational cost.

Related papers

• Robust Tickets Can Transfer Better: Drawing More Transferable Subnetworks in Transfer Learning [25.310066345466396]
  Transfer learning leverages feature representations of deep neural networks (DNNs) pretrained on source tasks with rich data to empower finetuning on downstream tasks. We propose a new transfer learning pipeline, which leverages our finding that robust tickets can transfer better, i.e.,works drawn with properly induced adversarial robustness can win better transferability over vanilla lottery ticketworks.
  arXiv  Detail & Related papers  (2023-04-24T05:44:42Z)
• The Role of Masking for Efficient Supervised Knowledge Distillation of Vision Transformers [14.467509261354458]
  In this paper, we develop a simple framework to reduce the supervision cost of ViT distillation. By masking input tokens, one can skip the computations associated with the masked tokens without requiring any change to teacher parameters or architecture. We find that masking patches with the lowest student attention scores is highly effective, saving up to 50% of teacher FLOPs without any drop in student accuracy.
  arXiv  Detail & Related papers  (2023-02-21T07:48:34Z)
• Robust Lottery Tickets for Pre-trained Language Models [57.14316619360376]
  We propose a novel method based on learning binary weight masks to identify robust tickets hidden in the original language models. Experimental results show the significant improvement of the proposed method over previous work on adversarial robustness evaluation.
  arXiv  Detail & Related papers  (2022-11-06T02:59:27Z)
• Dual Lottery Ticket Hypothesis [71.95937879869334]
  Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark. We propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH.
  arXiv  Detail & Related papers  (2022-03-08T18:06:26Z)
• You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership [87.13642800792077]
  Lottery ticket hypothesis (LTH) emerges as a promising framework to leverage a special sparse subnetwork. Main resource bottleneck of LTH is however the extraordinary cost to find the sparse mask of the winning ticket. Our setting adds a new dimension to the recently soaring interest in protecting against the intellectual property infringement of deep models.
  arXiv  Detail & Related papers  (2021-10-30T03:38:38Z)
• Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective [89.19516919095904]
  Winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. We make use of renormalization group theory, one of the most successful tools in theoretical physics. We leverage here to examine winning ticket universality in large scale lottery ticket experiments, as well as sheds new light on the success iterative magnitude pruning has found in the field of sparse machine learning.
  arXiv  Detail & Related papers  (2021-10-07T06:50:16Z)
• MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learning [65.52675802289775]
  We show that an uncertainty aware classifier can solve challenging reinforcement learning problems. We propose a novel method for computing the normalized maximum likelihood (NML) distribution. We show that the resulting algorithm has a number of intriguing connections to both count-based exploration methods and prior algorithms for learning reward functions.
  arXiv  Detail & Related papers  (2021-07-15T08:19:57Z)
• The Lottery Ticket Hypothesis for Object Recognition [39.186511997089575]
  Lottery Ticket Hypothesis states that deep networks trained on large datasets contain smaller neuralworks that achieve on par performance as the dense networks. We provide guidance on how to find lottery tickets with up to 80% overall sparsity on different sub-tasks without incurring any drop in performance.
  arXiv  Detail & Related papers  (2020-12-08T18:59:13Z)
• Towards Practical Lottery Ticket Hypothesis for Adversarial Training [78.30684998080346]
  We show there exists a subset of the aforementioned sub-networks that converge significantly faster during the training process. As a practical application of our findings, we demonstrate that such sub-networks can help in cutting down the total time of adversarial training.
  arXiv  Detail & Related papers  (2020-03-06T03:11:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.