SYMPOL: Symbolic Tree-Based On-Policy Reinforcement Learning

• URL: http://arxiv.org/abs/2408.08761v1
• Date: Fri, 16 Aug 2024 14:04:40 GMT
• Title: SYMPOL: Symbolic Tree-Based On-Policy Reinforcement Learning
• Authors: Sascha Marton, Tim Grams, Florian Vogt, Stefan Lüdtke, Christian Bartelt, Heiner Stuckenschmidt,
• Abstract summary: We introduce SYMPOL, a novel method for SYMbolic tree-based on-POLicy RL. SYMPOL employs a tree-based model integrated with a policy gradient method, enabling the agent to learn and adapt its actions. We evaluate SYMPOL on a set of benchmark RL tasks, demonstrating its superiority over alternative tree-based RL approaches.
• Score: 9.035959289139102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) has seen significant success across various domains, but its adoption is often limited by the black-box nature of neural network policies, making them difficult to interpret. In contrast, symbolic policies allow representing decision-making strategies in a compact and interpretable way. However, learning symbolic policies directly within on-policy methods remains challenging. In this paper, we introduce SYMPOL, a novel method for SYMbolic tree-based on-POLicy RL. SYMPOL employs a tree-based model integrated with a policy gradient method, enabling the agent to learn and adapt its actions while maintaining a high level of interpretability. We evaluate SYMPOL on a set of benchmark RL tasks, demonstrating its superiority over alternative tree-based RL approaches in terms of performance and interpretability. To the best of our knowledge, this is the first method, that allows a gradient-based end-to-end learning of interpretable, axis-aligned decision trees on-policy. Therefore, SYMPOL can become the foundation for a new class of interpretable RL based on decision trees. Our implementation is available under: https://github.com/s-marton/SYMPOL

Related papers

• In Search of Trees: Decision-Tree Policy Synthesis for Black-Box Systems via Search [6.74890780471356]
  We present an approach to synthesise optimal decision-tree policies given a black-box environment and specification. Our approach is a specialised search algorithm which systematically explores the space of decision trees under the given discretisation.
  arXiv  Detail & Related papers  (2024-09-05T05:51:42Z)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients [62.81324245896716]
  Policy gradient (PG) methods are successful approaches to deal with continuous reinforcement learning (RL) problems. In common practice, convergence (hyper)policies are learned only to deploy their deterministic version. We show how to tune the exploration level used for learning to optimize the trade-off between the sample complexity and the performance of the deployed deterministic policy.
  arXiv  Detail & Related papers  (2024-05-03T16:45:15Z)
• Distilling Reinforcement Learning Policies for Interpretable Robot Locomotion: Gradient Boosting Machines and Symbolic Regression [53.33734159983431]
  This paper introduces a novel approach to distill neural RL policies into more interpretable forms. We train expert neural network policies using RL and distill them into (i) GBMs, (ii) EBMs, and (iii) symbolic policies.
  arXiv  Detail & Related papers  (2024-03-21T11:54:45Z)
• Nash Learning from Human Feedback [86.09617990412941]
  We introduce an alternative pipeline for the fine-tuning of large language models using pairwise human feedback. We term this approach Nash learning from human feedback (NLHF) We present a novel algorithmic solution, Nash-MD, founded on the principles of mirror descent.
  arXiv  Detail & Related papers  (2023-12-01T19:26:23Z)
• Efficient Symbolic Policy Learning with Differentiable Symbolic Expression [30.855457609733637]
  We propose an efficient gradient-based learning method that learns the symbolic policy from scratch in an end-to-end way. In addition, in contrast with previous symbolic policies which only work in single-task RL because of complexity, we expand ESPL on meta-RL to generate symbolic policies for unseen tasks.
  arXiv  Detail & Related papers  (2023-11-02T03:27:51Z)
• When is Agnostic Reinforcement Learning Statistically Tractable? [76.1408672715773]
  A new complexity measure, called the emphspanning capacity, depends solely on the set $Pi$ and is independent of the MDP dynamics. We show there exists a policy class $Pi$ with a bounded spanning capacity that requires a superpolynomial number of samples to learn. This reveals a surprising separation for learnability between generative access and online access models.
  arXiv  Detail & Related papers  (2023-10-09T19:40:54Z)
• A Surprisingly Simple Continuous-Action POMDP Solver: Lazy Cross-Entropy Search Over Policy Trees [5.250288418639076]
  We propose an online POMDP solver called Lazy Cross-Entropy Search Over Policy Trees (LCEOPT) At each planning step, our method uses a novel lazy Cross-Entropy method to search the space of policy trees. Our method is surprisingly simple as compared to existing state-of-the-art methods, yet empirically outperforms them on several continuous-action POMDP problems.
  arXiv  Detail & Related papers  (2023-05-14T03:12:53Z)
• Deep Explainable Relational Reinforcement Learning: A Neuro-Symbolic Approach [18.38878415765146]
  We propose Explainable Reinforcement Learning (DERRL), a framework that exploits the best of both -- neural and symbolic worlds. DERRL combines relational representations and constraints from symbolic planning with deep learning to extract interpretable policies. These policies are in the form of logical rules that explain how each decision (or action) is arrived at.
  arXiv  Detail & Related papers  (2023-04-17T15:11:40Z)
• Policy Gradient for Reinforcement Learning with General Utilities [50.65940899590487]
  In Reinforcement Learning (RL), the goal of agents is to discover an optimal policy that maximizes the expected cumulative rewards. Many supervised and unsupervised RL problems are not covered in the Linear RL framework. We derive the policy gradient theorem for RL with general utilities.
  arXiv  Detail & Related papers  (2022-10-03T14:57:46Z)
• Continuous Action Reinforcement Learning from a Mixture of Interpretable Experts [35.80418547105711]
  We propose a policy scheme that retains a complex function approxor for its internal value predictions but constrains the policy to have a concise, hierarchical, and human-readable structure. The main technical contribution of the paper is to address the challenges introduced by this non-differentiable state selection procedure.
  arXiv  Detail & Related papers  (2020-06-10T16:02:08Z)
• Tree-Structured Policy based Progressive Reinforcement Learning for Temporally Language Grounding in Video [128.08590291947544]
  Temporally language grounding in untrimmed videos is a newly-raised task in video understanding. Inspired by human's coarse-to-fine decision-making paradigm, we formulate a novel Tree-Structured Policy based Progressive Reinforcement Learning framework.
  arXiv  Detail & Related papers  (2020-01-18T15:08:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.