Hierarchical State Abstraction Based on Structural Information Principles

• URL: http://arxiv.org/abs/2304.12000v1
• Date: Mon, 24 Apr 2023 11:06:52 GMT
• Title: Hierarchical State Abstraction Based on Structural Information Principles
• Authors: Xianghua Zeng, Hao Peng, Angsheng Li, Chunyang Liu, Lifang He, Philip S. Yu
• Abstract summary: We propose a novel mathematical Structural Information principles-based State Abstraction framework, namely SISA, from the information-theoretic perspective. SISA is a general framework that can be flexibly integrated with different representation-learning objectives to improve their performances further.
• Score: 70.24495170921075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: State abstraction optimizes decision-making by ignoring irrelevant environmental information in reinforcement learning with rich observations. Nevertheless, recent approaches focus on adequate representational capacities resulting in essential information loss, affecting their performances on challenging tasks. In this article, we propose a novel mathematical Structural Information principles-based State Abstraction framework, namely SISA, from the information-theoretic perspective. Specifically, an unsupervised, adaptive hierarchical state clustering method without requiring manual assistance is presented, and meanwhile, an optimal encoding tree is generated. On each non-root tree node, a new aggregation function and condition structural entropy are designed to achieve hierarchical state abstraction and compensate for sampling-induced essential information loss in state abstraction. Empirical evaluations on a visual gridworld domain and six continuous control benchmarks demonstrate that, compared with five SOTA state abstraction approaches, SISA significantly improves mean episode reward and sample efficiency up to 18.98 and 44.44%, respectively. Besides, we experimentally show that SISA is a general framework that can be flexibly integrated with different representation-learning objectives to improve their performances further.

Related papers

• Effective Reinforcement Learning Based on Structural Information Principles [19.82391136775341]
  We propose a novel and general Structural Information principles-based framework for effective Decision-Making, namely SIDM. SIDM can be flexibly incorporated into various single-agent and multi-agent RL algorithms, enhancing their performance.
  arXiv  Detail & Related papers  (2024-04-15T13:02:00Z)
• Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning [83.41487567765871]
  Skipper is a model-based reinforcement learning framework. It automatically generalizes the task given into smaller, more manageable subtasks. It enables sparse decision-making and focused abstractions on the relevant parts of the environment.
  arXiv  Detail & Related papers  (2023-09-30T02:25:18Z)
• A Direct Approximation of AIXI Using Logical State Abstractions [6.570488724773507]
  We propose a practical integration of logical state abstraction with AIXI, a Bayesian optimality notion for reinforcement learning agents. We address the problem of selecting the right subset of features to form state abstractions. Exact Bayesian model learning is then achieved using a suitable generalisation of Context Tree Weighting over abstract state sequences.
  arXiv  Detail & Related papers  (2022-10-13T11:30:56Z)
• Causal Dynamics Learning for Task-Independent State Abstraction [61.707048209272884]
  We introduce Causal Dynamics Learning for Task-Independent State Abstraction (CDL) CDL learns a theoretically proved causal dynamics model that removes unnecessary dependencies between state variables and the action. A state abstraction can then be derived from the learned dynamics.
  arXiv  Detail & Related papers  (2022-06-27T17:02:53Z)
• MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspective [57.19660234992812]
  NER model has achieved promising performance on standard NER benchmarks. Recent studies show that previous approaches may over-rely on entity mention information, resulting in poor performance on out-of-vocabulary (OOV) entity recognition. We propose MINER, a novel NER learning framework, to remedy this issue from an information-theoretic perspective.
  arXiv  Detail & Related papers  (2022-04-09T05:18:20Z)
• Dynamic probabilistic logic models for effective abstractions in RL [35.54018388244684]
  RePReL is a hierarchical framework that leverages a relational planner to provide useful state abstractions for learning. Our experiments show that RePReL not only achieves better performance and efficient learning on the task at hand but also demonstrates better generalization to unseen tasks.
  arXiv  Detail & Related papers  (2021-10-15T18:53:04Z)
• Action-Sufficient State Representation Learning for Control with Structural Constraints [21.47086290736692]
  In this paper, we focus on partially observable environments and propose to learn a minimal set of state representations that capture sufficient information for decision-making. We build a generative environment model for the structural relationships among variables in the system and present a principled way to characterize ASRs. Our empirical results on CarRacing and VizDoom demonstrate a clear advantage of learning and using ASRs for policy learning.
  arXiv  Detail & Related papers  (2021-10-12T03:16:26Z)
• CDEvalSumm: An Empirical Study of Cross-Dataset Evaluation for Neural Summarization Systems [121.78477833009671]
  We investigate the performance of different summarization models under a cross-dataset setting. A comprehensive study of 11 representative summarization systems on 5 datasets from different domains reveals the effect of model architectures and generation ways.
  arXiv  Detail & Related papers  (2020-10-11T02:19:15Z)
• Randomized Value Functions via Posterior State-Abstraction Sampling [21.931580762349096]
  We argue that an agent seeking out latent task structure must explicitly represent and maintain its uncertainty over that structure. We introduce a practical algorithm for doing this using two posterior distributions over state abstractions and abstract-state values. In empirically validating our approach, we find that substantial performance gains lie in the multi-task setting.
  arXiv  Detail & Related papers  (2020-10-05T23:04:18Z)
• A Dependency Syntactic Knowledge Augmented Interactive Architecture for End-to-End Aspect-based Sentiment Analysis [73.74885246830611]
  We propose a novel dependency syntactic knowledge augmented interactive architecture with multi-task learning for end-to-end ABSA. This model is capable of fully exploiting the syntactic knowledge (dependency relations and types) by leveraging a well-designed Dependency Relation Embedded Graph Convolutional Network (DreGcn) Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-04T14:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.