Learning telic-controllable state representations

• URL: http://arxiv.org/abs/2406.14476v3
• Date: Mon, 04 Aug 2025 20:02:53 GMT
• Title: Learning telic-controllable state representations
• Authors: Nadav Amir, Stas Tiomkin,
• Abstract summary: We present a computational framework for state representation learning in bounded agents.<n>We introduce the concept of telic-controllability to characterize the tradeoff between the granularity of a telic state representation and the policy complexity required to reach all telic states.<n>Our framework highlights the role of deliberate ignorance -- knowing what to ignore -- for learning state representations that balance goal flexibility and cognitive complexity.
• Score: 3.4530027457862
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computational models of purposeful behavior comprise both descriptive and prescriptive aspects, used respectively to ascertain and evaluate situations in the world. In reinforcement learning, prescriptive reward functions are assumed to depend on predefined and fixed descriptive state representations. Alternatively, these two aspects may emerge interdependently: goals can shape the acquired state representations and vice versa. Here, we present a computational framework for state representation learning in bounded agents, where descriptive and prescriptive aspects are coupled through the notion of goal-directed, or telic, states. We introduce the concept of telic-controllability to characterize the tradeoff between the granularity of a telic state representation and the policy complexity required to reach all telic states. We propose an algorithm for learning telic-controllable state representations, illustrating it using a simulated navigation task. Our framework highlights the role of deliberate ignorance -- knowing what to ignore -- for learning state representations that balance goal flexibility and cognitive complexity.

Related papers

• Learning with Language-Guided State Abstractions [58.199148890064826]
  Generalizable policy learning in high-dimensional observation spaces is facilitated by well-designed state representations. Our method, LGA, uses a combination of natural language supervision and background knowledge from language models to automatically build state representations tailored to unseen tasks. Experiments on simulated robotic tasks show that LGA yields state abstractions similar to those designed by humans, but in a fraction of the time.
  arXiv  Detail & Related papers  (2024-02-28T23:57:04Z)
• Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning [57.67629402360924]
  We introduce the Partially Supervised Reinforcement Learning (PSRL) framework. At the heart of PSRL is the fusion of both supervised and unsupervised learning. We show that PSRL offers a potent balance, enhancing model interpretability while preserving, and often significantly outperforming, the performance benchmarks set by traditional methods.
  arXiv  Detail & Related papers  (2024-02-14T16:23:23Z)
• Provable Representation with Efficient Planning for Partial Observable Reinforcement Learning [74.67655210734338]
  In most real-world reinforcement learning applications, state information is only partially observable, which breaks the Markov decision process assumption. We develop a representation-based perspective that leads to a coherent framework and tractable algorithmic approach for practical reinforcement learning from partial observations. We empirically demonstrate the proposed algorithm can surpass state-of-the-art performance with partial observations across various benchmarks.
  arXiv  Detail & Related papers  (2023-11-20T23:56:58Z)
• Inverse Decision Modeling: Learning Interpretable Representations of Behavior [72.80902932543474]
  We develop an expressive, unifying perspective on inverse decision modeling. We use this to formalize the inverse problem (as a descriptive model) We illustrate how this structure enables learning (interpretable) representations of (bounded) rationality.
  arXiv  Detail & Related papers  (2023-10-28T05:05:01Z)
• State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping [3.4777703321218225]
  This work examines the effect of various representations in incentivizing the agent to solve a specific robotic task. A continuum of state representations is defined, starting from hand-crafted numerical states to encoded image-based representations. The effects of each representation on the ability of the agent to solve the task in simulation and the transferability of the learned policy to the real robot are examined.
  arXiv  Detail & Related papers  (2023-09-21T11:41:22Z)
• Hierarchical State Abstraction Based on Structural Information Principles [70.24495170921075]
  We propose a novel mathematical Structural Information principles-based State Abstraction framework, namely SISA, from the information-theoretic perspective. SISA is a general framework that can be flexibly integrated with different representation-learning objectives to improve their performances further.
  arXiv  Detail & Related papers  (2023-04-24T11:06:52Z)
• Neural Distillation as a State Representation Bottleneck in Reinforcement Learning [4.129225533930966]
  We argue that distillation can be used to learn a state representation displaying favorable characteristics. We first evaluate these criteria and verify the contribution of distillation on state representation on a toy environment based on the standard inverted pendulum problem.
  arXiv  Detail & Related papers  (2022-10-05T13:00:39Z)
• Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning [71.52722621691365]
  Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. We propose a new form of state abstraction called goal-conditioned bisimulation. We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-27T17:00:11Z)
• On the Generalization of Representations in Reinforcement Learning [32.303656009679045]
  We provide an informative bound on the generalization error arising from a specific state representation. Our bound applies to any state representation and quantifies the natural tension between representations that generalize well and those that approximate well.
  arXiv  Detail & Related papers  (2022-03-01T15:22:09Z)
• Interpretable Reinforcement Learning with Multilevel Subgoal Discovery [77.34726150561087]
  We propose a novel Reinforcement Learning model for discrete environments. In the model, an agent learns information about environment in the form of probabilistic rules. No reward function is required for learning; an agent only needs to be given a primary goal to achieve.
  arXiv  Detail & Related papers  (2022-02-15T14:04:44Z)
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning [120.38381203153159]
  Reinforcement learning can train policies that effectively perform complex tasks. For long-horizon tasks, the performance of these methods degrades with horizon, often necessitating reasoning over and composing lower-level skills. We propose Value Function Spaces: a simple approach that produces such a representation by using the value functions corresponding to each lower-level skill.
  arXiv  Detail & Related papers  (2021-11-04T22:46:16Z)
• Desiderata for Representation Learning: A Causal Perspective [104.3711759578494]
  We take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.
  arXiv  Detail & Related papers  (2021-09-08T17:33:54Z)
• A Tutorial on Learning Disentangled Representations in the Imaging Domain [13.320565017546985]
  Disentangled representation learning has been proposed as an approach to learning general representations. A good general representation can be readily fine-tuned for new target tasks using modest amounts of data. Disentangled representations can offer model explainability and can help us understand the underlying causal relations of the factors of variation.
  arXiv  Detail & Related papers  (2021-08-26T21:44:10Z)
• Towards Learning Controllable Representations of Physical Systems [9.088303226909279]
  Learned representations of dynamical systems reduce dimensionality, potentially supporting downstream reinforcement learning (RL) We consider the relationship between the true state and the corresponding representations, proposing that ideally each representation corresponds to a unique state. These metrics are shown to predict reinforcement learning performance in a simulated peg-in-hole task when comparing variants of autoencoder-based representations.
  arXiv  Detail & Related papers  (2020-11-16T17:15:57Z)
• An Overview of Natural Language State Representation for Reinforcement Learning [17.285206913252786]
  A suitable state representation is a fundamental part of the learning process in Reinforcement Learning. This survey outlines the strategies used in the literature to build natural language state representations.
  arXiv  Detail & Related papers  (2020-07-19T20:15:55Z)
• Fairness by Learning Orthogonal Disentangled Representations [50.82638766862974]
  We propose a novel disentanglement approach to invariant representation problem. We enforce the meaningful representation to be agnostic to sensitive information by entropy. The proposed approach is evaluated on five publicly available datasets.
  arXiv  Detail & Related papers  (2020-03-12T11:09:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.