Interference and Generalization in Temporal Difference Learning
- URL: http://arxiv.org/abs/2003.06350v1
- Date: Fri, 13 Mar 2020 15:49:58 GMT
- Title: Interference and Generalization in Temporal Difference Learning
- Authors: Emmanuel Bengio, Joelle Pineau, Doina Precup
- Abstract summary: We study the link between generalization and interference in temporal-difference (TD) learning.
We find that TD easily leads to low-interference, under-generalizing parameters, while the effect seems reversed in supervised learning.
- Score: 86.31598155056035
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We study the link between generalization and interference in
temporal-difference (TD) learning. Interference is defined as the inner product
of two different gradients, representing their alignment. This quantity emerges
as being of interest from a variety of observations about neural networks,
parameter sharing and the dynamics of learning. We find that TD easily leads to
low-interference, under-generalizing parameters, while the effect seems
reversed in supervised learning. We hypothesize that the cause can be traced
back to the interplay between the dynamics of interference and bootstrapping.
This is supported empirically by several observations: the negative
relationship between the generalization gap and interference in TD, the
negative effect of bootstrapping on interference and the local coherence of
targets, and the contrast between the propagation rate of information in TD(0)
versus TD($\lambda$) and regression tasks such as Monte-Carlo policy
evaluation. We hope that these new findings can guide the future discovery of
better bootstrapping methods.
Related papers
- Two-Phase Dynamics of Interactions Explains the Starting Point of a DNN Learning Over-Fitted Features [68.3512123520931]
We investigate the dynamics of a deep neural network (DNN) learning interactions.
In this paper, we discover the DNN learns interactions in two phases.
The first phase mainly penalizes interactions of medium and high orders, and the second phase mainly learns interactions of gradually increasing orders.
arXiv Detail & Related papers (2024-05-16T17:13:25Z) - Controllable Relation Disentanglement for Few-Shot Class-Incremental Learning [82.79371269942146]
We propose to tackle FewShot Class-Incremental Learning (FSCIL) from a new perspective, i.e., relation disentanglement.
The challenge of disentangling spurious correlations lies in the poor controllability of FSCIL.
We propose a new simple-yeteffective method, called ConTrollable Relation-disentang FewShot Class-Incremental Learning (CTRL-FSCIL)
arXiv Detail & Related papers (2024-03-17T03:16:59Z) - Distal Interference: Exploring the Limits of Model-Based Continual
Learning [0.0]
Continual learning is hindered by catastrophic interference or forgetting.
Despite their practical success, artificial neural networks (ANNs) are prone to catastrophic interference.
It is conjectured that continual learning with complexity models requires augmentation of the training data or algorithm.
arXiv Detail & Related papers (2024-02-13T07:07:37Z) - Interactive Autonomous Navigation with Internal State Inference and
Interactivity Estimation [58.21683603243387]
We propose three auxiliary tasks with relational-temporal reasoning and integrate them into the standard Deep Learning framework.
These auxiliary tasks provide additional supervision signals to infer the behavior patterns other interactive agents.
Our approach achieves robust and state-of-the-art performance in terms of standard evaluation metrics.
arXiv Detail & Related papers (2023-11-27T18:57:42Z) - Measuring and Mitigating Interference in Reinforcement Learning [30.38857177546063]
Catastrophic interference is common in many network-based learning systems.
We provide a definition and novel measure of interference for value-based reinforcement learning methods.
arXiv Detail & Related papers (2023-07-10T20:20:20Z) - Neighborhood Adaptive Estimators for Causal Inference under Network
Interference [152.4519491244279]
We consider the violation of the classical no-interference assumption, meaning that the treatment of one individuals might affect the outcomes of another.
To make interference tractable, we consider a known network that describes how interference may travel.
We study estimators for the average direct treatment effect on the treated in such a setting.
arXiv Detail & Related papers (2022-12-07T14:53:47Z) - Investigating the Role of Negatives in Contrastive Representation
Learning [59.30700308648194]
Noise contrastive learning is a popular technique for unsupervised representation learning.
We focus on disambiguating the role of one of these parameters: the number of negative examples.
We find that the results broadly agree with our theory, while our vision experiments are murkier with performance sometimes even being insensitive to the number of negatives.
arXiv Detail & Related papers (2021-06-18T06:44:16Z) - Interpreting Deep Neural Networks with Relative Sectional Propagation by
Analyzing Comparative Gradients and Hostile Activations [37.11665902583138]
We propose a new attribution method, Relative Sectional Propagation (RSP), for decomposing the output predictions of Deep Neural Networks (DNNs)
We define hostile factor as an element that interferes with finding the attributions of the target and propagates it in a distinguishable way to overcome the non-suppressed nature of activated neurons.
Our method makes it possible to decompose the predictions of DNNs with clearer class-discriminativeness and detailed elucidations of activation neurons compared to the conventional attribution methods.
arXiv Detail & Related papers (2020-12-07T03:11:07Z) - A Unified Approach to Interpreting and Boosting Adversarial
Transferability [42.33597623865435]
In this paper, we use the interaction inside adversarial perturbations to explain and boost the adversarial transferability.
We prove and prove the negative correlation between the adversarial transferability and the interaction inside adversarial perturbations.
We propose to penalize interactions during the attacking process, which significantly improves the adversarial transferability.
arXiv Detail & Related papers (2020-10-08T15:19:22Z) - Towards a practical measure of interference for reinforcement learning [37.1734757628306]
Catastrophic interference is common in many network-based learning systems.
We provide a definition of interference for control in reinforcement learning.
Our new interference measure allows us to ask novel scientific questions about commonly used deep learning architectures.
arXiv Detail & Related papers (2020-07-07T22:02:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.