Leveraging class abstraction for commonsense reinforcement learning via residual policy gradient methods

• URL: http://arxiv.org/abs/2201.12126v1
• Date: Fri, 28 Jan 2022 14:03:49 GMT
• Title: Leveraging class abstraction for commonsense reinforcement learning via residual policy gradient methods
• Authors: Niklas H\"opner, Ilaria Tiddi, Herke van Hoof
• Abstract summary: We propose to use the subclass relationships present in open-source knowledge graphs to abstract away from specific objects. We develop a residual policy gradient method that is able to integrate knowledge across different abstraction levels in the class hierarchy.
• Score: 23.199881381599617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Enabling reinforcement learning (RL) agents to leverage a knowledge base while learning from experience promises to advance RL in knowledge intensive domains. However, it has proven difficult to leverage knowledge that is not manually tailored to the environment. We propose to use the subclass relationships present in open-source knowledge graphs to abstract away from specific objects. We develop a residual policy gradient method that is able to integrate knowledge across different abstraction levels in the class hierarchy. Our method results in improved sample efficiency and generalisation to unseen objects in commonsense games, but we also investigate failure modes, such as excessive noise in the extracted class knowledge or environments with little class structure.

Related papers

• Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models [33.504700578933424]
  Low sample efficiency is an enduring challenge of reinforcement learning (RL) We introduce a framework that harnesses large language models to extract background knowledge of an environment. Our experiments show that these methods achieve significant sample efficiency improvements in a spectrum of downstream tasks.
  arXiv  Detail & Related papers  (2024-07-04T14:33:47Z)
• KEHRL: Learning Knowledge-Enhanced Language Representations with Hierarchical Reinforcement Learning [32.086825891769585]
  Knowledge-enhanced pre-trained language models (KEPLMs) leverage relation triples from knowledge graphs (KGs) Previous works treat knowledge enhancement as two independent operations, i.e., knowledge injection and knowledge integration. This paper jointly addresses the problems of detecting positions for knowledge injection and integrating external knowledge into the model in order to avoid injecting inaccurate or irrelevant knowledge.
  arXiv  Detail & Related papers  (2024-06-24T07:32:35Z)
• InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration [61.554209059971576]
  Large Language Models (LLMs) have shown remarkable open-generation capabilities across diverse domains. Injecting new knowledge poses the risk of forgetting previously acquired knowledge. We propose a novel Infuser-Guided Knowledge Integration framework.
  arXiv  Detail & Related papers  (2024-02-18T03:36:26Z)
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning [63.58935783293342]
  Causal Bisimulation Modeling (CBM) is a method that learns the causal relationships in the dynamics and reward functions for each task to derive a minimal, task-specific abstraction. CBM's learned implicit dynamics models identify the underlying causal relationships and state abstractions more accurately than explicit ones.
  arXiv  Detail & Related papers  (2024-01-23T05:43:15Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation [68.13453771001522]
  We propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings. We conduct extensive experiments and evaluate our model on large-scale real-world data.
  arXiv  Detail & Related papers  (2023-06-14T13:07:48Z)
• Inapplicable Actions Learning for Knowledge Transfer in Reinforcement Learning [3.194414753332705]
  We show that learning inapplicable actions greatly improves the sample efficiency of RL algorithms. Thanks to the transferability of the knowledge acquired, it can be reused in other tasks and domains to make the learning process more efficient.
  arXiv  Detail & Related papers  (2022-11-28T17:45:39Z)
• Implicit Offline Reinforcement Learning via Supervised Learning [83.8241505499762]
  Offline Reinforcement Learning (RL) via Supervised Learning is a simple and effective way to learn robotic skills from a dataset collected by policies of different expertise levels. We show how implicit models can leverage return information and match or outperform explicit algorithms to acquire robotic skills from fixed datasets.
  arXiv  Detail & Related papers  (2022-10-21T21:59:42Z)
• Ontology-enhanced Prompt-tuning for Few-shot Learning [41.51144427728086]
  Few-shot Learning is aimed to make predictions based on a limited number of samples. Structured data such as knowledge graphs and ontology libraries has been leveraged to benefit the few-shot setting in various tasks.
  arXiv  Detail & Related papers  (2022-01-27T05:41:36Z)
• WordCraft: An Environment for Benchmarking Commonsense Agents [107.20421897619002]
  We propose WordCraft, an RL environment based on Little Alchemy 2. This lightweight environment is fast to run and built upon entities and relations inspired by real-world semantics.
  arXiv  Detail & Related papers  (2020-07-17T18:40:46Z)
• Contextual Policy Transfer in Reinforcement Learning Domains via Deep Mixtures-of-Experts [24.489002406693128]
  We introduce a novel mixture-of-experts formulation for learning state-dependent beliefs over source task dynamics. We show how this model can be incorporated into standard policy reuse frameworks.
  arXiv  Detail & Related papers  (2020-02-29T07:58:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.