On the Generalization of Representations in Reinforcement Learning

• URL: http://arxiv.org/abs/2203.00543v1
• Date: Tue, 1 Mar 2022 15:22:09 GMT
• Title: On the Generalization of Representations in Reinforcement Learning
• Authors: Charline Le Lan, Stephen Tu, Adam Oberman, Rishabh Agarwal, Marc G.Bellemare
• Abstract summary: We provide an informative bound on the generalization error arising from a specific state representation. Our bound applies to any state representation and quantifies the natural tension between representations that generalize well and those that approximate well.
• Score: 32.303656009679045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In reinforcement learning, state representations are used to tractably deal with large problem spaces. State representations serve both to approximate the value function with few parameters, but also to generalize to newly encountered states. Their features may be learned implicitly (as part of a neural network) or explicitly (for example, the successor representation of \citet{dayan1993improving}). While the approximation properties of representations are reasonably well-understood, a precise characterization of how and when these representations generalize is lacking. In this work, we address this gap and provide an informative bound on the generalization error arising from a specific state representation. This bound is based on the notion of effective dimension which measures the degree to which knowing the value at one state informs the value at other states. Our bound applies to any state representation and quantifies the natural tension between representations that generalize well and those that approximate well. We complement our theoretical results with an empirical survey of classic representation learning methods from the literature and results on the Arcade Learning Environment, and find that the generalization behaviour of learned representations is well-explained by their effective dimension.

Related papers

• Learning telic-controllable state representations [3.072340427031969]
  We present a novel computational framework for state representation learning in bounded agents. Our work advances a unified theoretical perspective on goal-directed state representation learning in natural and artificial agents.
  arXiv  Detail & Related papers  (2024-06-20T16:38:25Z)
• Representations as Language: An Information-Theoretic Framework for Interpretability [7.2129390689756185]
  Large scale neural models show impressive performance across a wide array of linguistic tasks. Despite this they remain, largely, black-boxes, inducing vector-representations of their input that prove difficult to interpret. We introduce a novel approach to interpretability that looks at the mapping a model learns from sentences to representations as a kind of language in its own right.
  arXiv  Detail & Related papers  (2024-06-04T16:14:00Z)
• Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [59.138470433237615]
  We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning. We show that systematically controlled metrics are strongly predictive of generalization performance. This work informs an important direction towards quality-enhancing the data diversity or balance to scaling up the absolute size.
  arXiv  Detail & Related papers  (2024-03-25T03:18:39Z)
• Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning [57.67629402360924]
  We introduce the Partially Supervised Reinforcement Learning (PSRL) framework. At the heart of PSRL is the fusion of both supervised and unsupervised learning. We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
  arXiv  Detail & Related papers  (2024-02-14T16:23:23Z)
• Pointwise Representational Similarity [14.22332335495585]
  Pointwise Normalized Kernel Alignment (PNKA) is a measure that quantifies how similarly an individual input is represented in two representation spaces. We show how PNKA can be leveraged to develop a deeper understanding of (a) the input examples that are likely to be misclassified, (b) the concepts encoded by (individual) neurons in a layer, and (c) the effects of fairness interventions on learned representations.
  arXiv  Detail & Related papers  (2023-05-30T09:40:08Z)
• Desiderata for Representation Learning: A Causal Perspective [104.3711759578494]
  We take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.
  arXiv  Detail & Related papers  (2021-09-08T17:33:54Z)
• Metrics and continuity in reinforcement learning [34.10996560464196]
  We introduce a unified formalism for defining topologies through the lens of metrics. We establish a hierarchy amongst these metrics and demonstrate their theoretical implications on the Markov Decision Process. We complement our theoretical results with empirical evaluations showcasing the differences between the metrics considered.
  arXiv  Detail & Related papers  (2021-02-02T14:30:41Z)
• Towards Learning Controllable Representations of Physical Systems [9.088303226909279]
  Learned representations of dynamical systems reduce dimensionality, potentially supporting downstream reinforcement learning (RL) We consider the relationship between the true state and the corresponding representations, proposing that ideally each representation corresponds to a unique state. These metrics are shown to predict reinforcement learning performance in a simulated peg-in-hole task when comparing variants of autoencoder-based representations.
  arXiv  Detail & Related papers  (2020-11-16T17:15:57Z)
• Fairness by Learning Orthogonal Disentangled Representations [50.82638766862974]
  We propose a novel disentanglement approach to invariant representation problem. We enforce the meaningful representation to be agnostic to sensitive information by entropy. The proposed approach is evaluated on five publicly available datasets.
  arXiv  Detail & Related papers  (2020-03-12T11:09:15Z)
• Weakly-Supervised Disentanglement Without Compromises [53.55580957483103]
  Intelligent agents should be able to learn useful representations by observing changes in their environment. We model such observations as pairs of non-i.i.d. images sharing at least one of the underlying factors of variation. We show that only knowing how many factors have changed, but not which ones, is sufficient to learn disentangled representations.
  arXiv  Detail & Related papers  (2020-02-07T16:39:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.