$\texttt{FORM}$: Learning Expressive and Transferable First-Order Logic Reward Machines

• URL: http://arxiv.org/abs/2501.00364v2
• Date: Mon, 17 Feb 2025 10:52:47 GMT
• Title: $\texttt{FORM}$: Learning Expressive and Transferable First-Order Logic Reward Machines
• Authors: Leo Ardon, Daniel Furelos-Blanco, Roko Parac, Alessandra Russo,
• Abstract summary: Reward machines (RMs) are an effective approach for addressing non-Markovian rewards in reinforcement learning. We propose First-Order Reward Machines ($texttFORM$s), which use first-order logic to label edges. We show that $texttFORM$s can be effectively learnt for tasks where traditional RM learning approaches fail.
• Score: 48.36822060760614
• License:
• Abstract: Reward machines (RMs) are an effective approach for addressing non-Markovian rewards in reinforcement learning (RL) through finite-state machines. Traditional RMs, which label edges with propositional logic formulae, inherit the limited expressivity of propositional logic. This limitation hinders the learnability and transferability of RMs since complex tasks will require numerous states and edges. To overcome these challenges, we propose First-Order Reward Machines ($\texttt{FORM}$s), which use first-order logic to label edges, resulting in more compact and transferable RMs. We introduce a novel method for $\textbf{learning}$ $\texttt{FORM}$s and a multi-agent formulation for $\textbf{exploiting}$ them and facilitate their transferability, where multiple agents collaboratively learn policies for a shared $\texttt{FORM}$. Our experimental results demonstrate the scalability of $\texttt{FORM}$s with respect to traditional RMs. Specifically, we show that $\texttt{FORM}$s can be effectively learnt for tasks where traditional RM learning approaches fail. We also show significant improvements in learning speed and task transferability thanks to the multi-agent learning framework and the abstraction provided by the first-order language.

Related papers

• Reward Machine Inference for Robotic Manipulation [1.6135226672466307]
  Reward Machines (RMs) enhance RL's capability to train policies over extended time horizons. We introduce a novel LfD approach for learning RMs directly from visual demonstrations of robotic manipulation tasks. We validate our method on vision-based manipulation tasks, showing that the inferred RM accurately captures task structure and enables an RL agent to effectively learn an optimal policy.
  arXiv  Detail & Related papers  (2024-12-13T12:32:53Z)
• Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems [42.137278756052595]
  $texttAgentPrune$ can seamlessly integrate into mainstream multi-agent systems. textbf(I) integrates seamlessly into existing multi-agent frameworks with $28.1%sim72.8%downarrow$ token reduction. textbf(III) successfully defend against two types of agent-based adversarial attacks with $3.5%sim10.8%uparrow$ performance boost.
  arXiv  Detail & Related papers  (2024-10-03T14:14:31Z)
• Tabular Transfer Learning via Prompting LLMs [52.96022335067357]
  We propose a novel framework, Prompt to Transfer (P2T), that utilizes unlabeled (or heterogeneous) source data with large language models (LLMs) P2T identifies a column feature in a source dataset that is strongly correlated with a target task feature to create examples relevant to the target task, thus creating pseudo-demonstrations for prompts.
  arXiv  Detail & Related papers  (2024-08-09T11:30:52Z)
• Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation [43.32632163091792]
  Large Language Models (LLMs) have the ability to solve a variety of tasks, such as text summarization and mathematical questions. Due to high computational costs, the current trend is to use prompt instruction tuning to better adjust monolithic, pretrained LLMs for new -- but often individual -- downstream tasks. MoPs can simultaneously mitigate prompt training "interference" in multi-task, multi-source scenarios.
  arXiv  Detail & Related papers  (2023-10-04T14:11:12Z)
• Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing [38.883540444516605]
  DARPA launched the ShELL program, which aims to explore how experience sharing can benefit distributed lifelong learning agents. We conduct both theoretical and empirical research on distributed multi-task reinforcement learning (RL), where a group of $N$ agents collaboratively solves $M$ tasks. We propose an algorithm called DistMT-LSVI, where each agent independently learns $epsilon$-optimal policies for all $M$ tasks.
  arXiv  Detail & Related papers  (2023-07-11T22:58:53Z)
• Learning In-context Learning for Named Entity Recognition [54.022036267886214]
  Named entity recognition in real-world applications suffers from the diversity of entity types, the emergence of new entity types, and the lack of high-quality annotations. This paper proposes an in-context learning-based NER approach, which can effectively inject in-context NER ability into PLMs. We show that our method can effectively inject in-context NER ability into PLMs and significantly outperforms the PLMs+fine-tuning counterparts.
  arXiv  Detail & Related papers  (2023-05-18T15:31:34Z)
• Skill-Based Reinforcement Learning with Intrinsic Reward Matching [77.34726150561087]
  We present Intrinsic Reward Matching (IRM), which unifies task-agnostic skill pretraining and task-aware finetuning. IRM enables us to utilize pretrained skills far more effectively than previous skill selection methods.
  arXiv  Detail & Related papers  (2022-10-14T00:04:49Z)
• Hierarchies of Reward Machines [75.55324974788475]
  Reward machines (RMs) are a recent formalism for representing the reward function of a reinforcement learning task through a finite-state machine. We propose a formalism for further abstracting the subtask structure by endowing an RM with the ability to call other RMs.
  arXiv  Detail & Related papers  (2022-05-31T12:39:24Z)
• Learning to Ask Conversational Questions by Optimizing Levenshtein Distance [83.53855889592734]
  We introduce a Reinforcement Iterative Sequence Editing (RISE) framework that optimize the minimum Levenshtein distance (MLD) through explicit editing actions. RISE is able to pay attention to tokens that are related to conversational characteristics. Experimental results on two benchmark datasets show that RISE significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-30T08:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.