Fortuitous Forgetting in Connectionist Networks

• URL: http://arxiv.org/abs/2202.00155v1
• Date: Tue, 1 Feb 2022 00:15:58 GMT
• Title: Fortuitous Forgetting in Connectionist Networks
• Authors: Hattie Zhou, Ankit Vani, Hugo Larochelle, Aaron Courville
• Abstract summary: We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations.
• Score: 20.206607130719696
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Forgetting is often seen as an unwanted characteristic in both human and machine learning. However, we propose that forgetting can in fact be favorable to learning. We introduce "forget-and-relearn" as a powerful paradigm for shaping the learning trajectories of artificial neural networks. In this process, the forgetting step selectively removes undesirable information from the model, and the relearning step reinforces features that are consistently useful under different conditions. The forget-and-relearn framework unifies many existing iterative training algorithms in the image classification and language emergence literature, and allows us to understand the success of these algorithms in terms of the disproportionate forgetting of undesirable information. We leverage this understanding to improve upon existing algorithms by designing more targeted forgetting operations. Insights from our analysis provide a coherent view on the dynamics of iterative training in neural networks and offer a clear path towards performance improvements.

Related papers

Err

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.