On Reinforcement Learning, Effect Handlers, and the State Monad
- URL: http://arxiv.org/abs/2203.15426v1
- Date: Tue, 29 Mar 2022 10:46:58 GMT
- Title: On Reinforcement Learning, Effect Handlers, and the State Monad
- Authors: Ugo Dal Lago, Francesco Gavazzo and Alexis Ghyselen
- Abstract summary: We study the effects and handlers as a way to support decision-making abstractions in functional programs.
We express the underlying intelligence as a reinforcement learning algorithm implemented as a set of handlers for some of these operations.
We conclude by hinting at how type and effect handlers could ensure safety properties.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We study the algebraic effects and handlers as a way to support
decision-making abstractions in functional programs, whereas a user can ask a
learning algorithm to resolve choices without implementing the underlying
selection mechanism, and give a feedback by way of rewards. Differently from
some recently proposed approach to the problem based on the selection monad
[Abadi and Plotkin, LICS 2021], we express the underlying intelligence as a
reinforcement learning algorithm implemented as a set of handlers for some of
these algebraic operations, including those for choices and rewards. We show
how we can in practice use algebraic operations and handlers -- as available in
the programming language EFF -- to clearly separate the learning algorithm from
its environment, thus allowing for a good level of modularity. We then show how
the host language can be taken as a lambda-calculus with handlers, this way
showing what the essential linguistic features are. We conclude by hinting at
how type and effect systems could ensure safety properties, at the same time
pointing at some directions for further work.
Related papers
- How to Choose a Reinforcement-Learning Algorithm [29.76033485145459]
We streamline the process of choosing reinforcement-learning algorithms and action-distribution families.
We provide a structured overview of existing methods and their properties, as well as guidelines for when to choose which methods.
arXiv Detail & Related papers (2024-07-30T15:54:18Z) - Unveiling the Decision-Making Process in Reinforcement Learning with Genetic Programming [4.249842620609683]
Incomprehensibility is not an option for the use of (deep) reinforcement learning in the real world.
We propose a genetic programming framework to generate explanations for the decision-making process of already trained agents.
We show that we are comparable in performance but require much less hardware resources and computation time.
arXiv Detail & Related papers (2024-07-20T00:45:03Z) - A Pattern Language for Machine Learning Tasks [0.0]
We view objective functions as constraints on the behaviour of learners.
We develop a formal graphical language that allows us to separate the core tasks of a behaviour from its implementation details.
As proof-of-concept, we design a novel task that enables converting classifiers into generative models we call "manipulators"
arXiv Detail & Related papers (2024-07-02T16:50:27Z) - RLIF: Interactive Imitation Learning as Reinforcement Learning [56.997263135104504]
We show how off-policy reinforcement learning can enable improved performance under assumptions that are similar but potentially even more practical than those of interactive imitation learning.
Our proposed method uses reinforcement learning with user intervention signals themselves as rewards.
This relaxes the assumption that intervening experts in interactive imitation learning should be near-optimal and enables the algorithm to learn behaviors that improve over the potential suboptimal human expert.
arXiv Detail & Related papers (2023-11-21T21:05:21Z) - Explainable Data-Driven Optimization: From Context to Decision and Back
Again [76.84947521482631]
Data-driven optimization uses contextual information and machine learning algorithms to find solutions to decision problems with uncertain parameters.
We introduce a counterfactual explanation methodology tailored to explain solutions to data-driven problems.
We demonstrate our approach by explaining key problems in operations management such as inventory management and routing.
arXiv Detail & Related papers (2023-01-24T15:25:16Z) - How Fine-Tuning Allows for Effective Meta-Learning [50.17896588738377]
We present a theoretical framework for analyzing representations derived from a MAML-like algorithm.
We provide risk bounds on the best predictor found by fine-tuning via gradient descent, demonstrating that the algorithm can provably leverage the shared structure.
This separation result underscores the benefit of fine-tuning-based methods, such as MAML, over methods with "frozen representation" objectives in few-shot learning.
arXiv Detail & Related papers (2021-05-05T17:56:00Z) - Smart Choices and the Selection Monad [0.0]
Describing systems in terms of choices and their resulting costs and rewards offers the promise of freeing algorithm designers and programmers from specifying how those choices should be made.
We define two small languages that support decision-making abstractions: one with choices and rewards, and the other additionally with probabilities.
arXiv Detail & Related papers (2020-07-17T12:13:16Z) - Explaining Black Box Predictions and Unveiling Data Artifacts through
Influence Functions [55.660255727031725]
Influence functions explain the decisions of a model by identifying influential training examples.
We conduct a comparison between influence functions and common word-saliency methods on representative tasks.
We develop a new measure based on influence functions that can reveal artifacts in training data.
arXiv Detail & Related papers (2020-05-14T00:45:23Z) - Can We Learn Heuristics For Graphical Model Inference Using
Reinforcement Learning? [114.24881214319048]
We show that we can learn programs, i.e., policies, for solving inference in higher order Conditional Random Fields (CRFs) using reinforcement learning.
Our method solves inference tasks efficiently without imposing any constraints on the form of the potentials.
arXiv Detail & Related papers (2020-04-27T19:24:04Z) - Learning with Differentiable Perturbed Optimizers [54.351317101356614]
We propose a systematic method to transform operations into operations that are differentiable and never locally constant.
Our approach relies on perturbeds, and can be used readily together with existing solvers.
We show how this framework can be connected to a family of losses developed in structured prediction, and give theoretical guarantees for their use in learning tasks.
arXiv Detail & Related papers (2020-02-20T11:11:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.