Towards a practical measure of interference for reinforcement learning

• URL: http://arxiv.org/abs/2007.03807v1
• Date: Tue, 7 Jul 2020 22:02:00 GMT
• Title: Towards a practical measure of interference for reinforcement learning
• Authors: Vincent Liu, Adam White, Hengshuai Yao, Martha White
• Abstract summary: Catastrophic interference is common in many network-based learning systems. We provide a definition of interference for control in reinforcement learning. Our new interference measure allows us to ask novel scientific questions about commonly used deep learning architectures.
• Score: 37.1734757628306
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Catastrophic interference is common in many network-based learning systems, and many proposals exist for mitigating it. But, before we overcome interference we must understand it better. In this work, we provide a definition of interference for control in reinforcement learning. We systematically evaluate our new measures, by assessing correlation with several measures of learning performance, including stability, sample efficiency, and online and offline control performance across a variety of learning architectures. Our new interference measure allows us to ask novel scientific questions about commonly used deep learning architectures. In particular we show that target network frequency is a dominating factor for interference, and that updates on the last layer result in significantly higher interference than updates internal to the network. This new measure can be expensive to compute; we conclude with motivation for an efficient proxy measure and empirically demonstrate it is correlated with our definition of interference.

Related papers

• Doubly Robust Causal Effect Estimation under Networked Interference via Targeted Learning [24.63284452991301]
  We propose a doubly robust causal effect estimator under networked interference. Specifically, we generalize the targeted learning technique into the networked interference setting. We devise an end-to-end causal effect estimator by transforming the identified theoretical condition into a targeted loss.
  arXiv  Detail & Related papers  (2024-05-06T10:49:51Z)
• Graph Machine Learning based Doubly Robust Estimator for Network Causal Effects [17.44202934049009]
  We propose a novel methodology that combines graph machine learning approaches with the double machine learning framework. We demonstrate our method is accurate, robust, and scalable via an extensive simulation study.
  arXiv  Detail & Related papers  (2024-03-17T20:23:42Z)
• Measuring and Mitigating Interference in Reinforcement Learning [30.38857177546063]
  Catastrophic interference is common in many network-based learning systems. We provide a definition and novel measure of interference for value-based reinforcement learning methods.
  arXiv  Detail & Related papers  (2023-07-10T20:20:20Z)
• Deep Reinforcement Learning for Interference Management in UAV-based 3D Networks: Potentials and Challenges [137.47736805685457]
  We show that interference can still be effectively mitigated even without knowing its channel information. By harnessing interference, the proposed solutions enable the continued growth of civilian UAVs.
  arXiv  Detail & Related papers  (2023-05-11T18:06:46Z)
• Detecting Irregular Network Activity with Adversarial Learning and Expert Feedback [14.188603782159372]
  CAAD employs contrastive learning in an adversarial setup to learn effective representations of normal and anomalous behavior in wireless networks. We conduct rigorous performance comparisons of CAAD with several state-of-the-art anomaly detection techniques and verify that CAAD yields a mean performance improvement of 92.84%.
  arXiv  Detail & Related papers  (2022-10-01T20:44:14Z)
• Disentangling Transfer and Interference in Multi-Domain Learning [53.34444188552444]
  We study the conditions where interference and knowledge transfer occur in multi-domain learning. We propose new metrics disentangling interference and transfer and set up experimental protocols. We demonstrate our findings on the CIFAR-100, MiniPlaces, and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2021-07-02T01:30:36Z)
• Vulnerability Under Adversarial Machine Learning: Bias or Variance? [77.30759061082085]
  We investigate the effect of adversarial machine learning on the bias and variance of a trained deep neural network. Our analysis sheds light on why the deep neural networks have poor performance under adversarial perturbation. We introduce a new adversarial machine learning algorithm with lower computational complexity than well-known adversarial machine learning strategies.
  arXiv  Detail & Related papers  (2020-08-01T00:58:54Z)
• Interference and Generalization in Temporal Difference Learning [86.31598155056035]
  We study the link between generalization and interference in temporal-difference (TD) learning. We find that TD easily leads to low-interference, under-generalizing parameters, while the effect seems reversed in supervised learning.
  arXiv  Detail & Related papers  (2020-03-13T15:49:58Z)
• On Catastrophic Interference in Atari 2600 Games [104.61596014400892]
  We show that interference causes performance to plateau. We demonstrate performance boosts across architectures, learning algorithms and environments. A more refined analysis shows that learning one segment of a game often increases prediction errors elsewhere.
  arXiv  Detail & Related papers  (2020-02-28T00:55:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.