Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective

• URL: http://arxiv.org/abs/2110.03210v1
• Date: Thu, 7 Oct 2021 06:50:16 GMT
• Title: Universality of Deep Neural Network Lottery Tickets: A Renormalization Group Perspective
• Authors: William T. Redman, Tianlong Chen, Akshunna S. Dogra, Zhangyang Wang
• Abstract summary: Winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. We make use of renormalization group theory, one of the most successful tools in theoretical physics. We leverage here to examine winning ticket universality in large scale lottery ticket experiments, as well as sheds new light on the success iterative magnitude pruning has found in the field of sparse machine learning.
• Score: 89.19516919095904
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundational work on the Lottery Ticket Hypothesis has suggested an exciting corollary: winning tickets found in the context of one task can be transferred to similar tasks, possibly even across different architectures. While this has become of broad practical and theoretical interest, to date, there exists no detailed understanding of why winning ticket universality exists, or any way of knowing \textit{a priori} whether a given ticket can be transferred to a given task. To address these outstanding open questions, we make use of renormalization group theory, one of the most successful tools in theoretical physics. We find that iterative magnitude pruning, the method used for discovering winning tickets, is a renormalization group scheme. This opens the door to a wealth of existing numerical and theoretical tools, some of which we leverage here to examine winning ticket universality in large scale lottery ticket experiments, as well as sheds new light on the success iterative magnitude pruning has found in the field of sparse machine learning.

Related papers

• Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis [0.0]
  This thesis focuses on the Lottery Ticket Hypothesis (LTH) The LTH posits that within extensive Deep Neural Networks (DNNs), smaller, trainable "winning tickets" can achieve performance comparable to the full model. A key process in LTH, Iterative Magnitude Pruning (IMP), incrementally eliminates minimal weights, emulating stepwise learning in DNNs. In other words, we check if a winning ticket that works well for one specific problem could also work well for other, similar problems.
  arXiv  Detail & Related papers  (2023-08-06T14:36:57Z)
• Convolutional and Residual Networks Provably Contain Lottery Tickets [6.68999512375737]
  The Lottery Ticket Hypothesis continues to have a profound practical impact on the quest for deep neural networks that solve modern deep learning tasks at competitive performance. We prove that also modern architectures consisting of convolutional and residual layers that can be equipped with almost arbitrary activation functions can contain lottery tickets with high probability.
  arXiv  Detail & Related papers  (2022-05-04T22:20:01Z)
• Dual Lottery Ticket Hypothesis [71.95937879869334]
  Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark. We propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH.
  arXiv  Detail & Related papers  (2022-03-08T18:06:26Z)
• On the Existence of Universal Lottery Tickets [2.5234156040689237]
  Lottery ticket hypothesis conjectures existence of sparseworks of large randomly deep neural networks that can be successfully trained in isolation. Recent work has experimentally observed that some of these tickets can be practically reused across a variety of tasks, hinting at some form of universality. We formalize this concept and theoretically prove that not only do such universal tickets exist but they also do not require further training.
  arXiv  Detail & Related papers  (2021-11-22T12:12:00Z)
• You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership [87.13642800792077]
  Lottery ticket hypothesis (LTH) emerges as a promising framework to leverage a special sparse subnetwork. Main resource bottleneck of LTH is however the extraordinary cost to find the sparse mask of the winning ticket. Our setting adds a new dimension to the recently soaring interest in protecting against the intellectual property infringement of deep models.
  arXiv  Detail & Related papers  (2021-10-30T03:38:38Z)
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot? [90.50740705956638]
  We show concrete evidence to clarify whether the winning ticket exists across the major DNN architectures and/or applications. We find that the key training hyper parameters, such as learning rate and training epochs, are all highly correlated with whether and when the winning tickets can be identified.
  arXiv  Detail & Related papers  (2021-07-01T01:27:07Z)
• Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win [20.97456178983006]
  Lottery ticket hypothesis states that sparseworks exist in randomly dense networks that can be trained to the same accuracy as the dense network they reside in. We show that by using a training method that is stable with respect to linear mode connectivity, large networks can also be entirely rewound to initialization.
  arXiv  Detail & Related papers  (2021-06-13T10:06:06Z)
• The Elastic Lottery Ticket Hypothesis [106.79387235014379]
  Lottery Ticket Hypothesis raises keen attention to identifying sparse trainableworks or winning tickets. The most effective method to identify such winning tickets is still Iterative Magnitude-based Pruning. We propose a variety of strategies to tweak the winning tickets found from different networks of the same model family.
  arXiv  Detail & Related papers  (2021-03-30T17:53:45Z)
• Winning Lottery Tickets in Deep Generative Models [64.79920299421255]
  We show the existence of winning tickets in deep generative models such as GANs and VAEs. We also demonstrate the transferability of winning tickets across different generative models.
  arXiv  Detail & Related papers  (2020-10-05T21:45:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.