Automated Feature Selection for Inverse Reinforcement Learning

• URL: http://arxiv.org/abs/2403.15079v1
• Date: Fri, 22 Mar 2024 10:05:21 GMT
• Title: Automated Feature Selection for Inverse Reinforcement Learning
• Authors: Daulet Baimukashev, Gokhan Alcan, Ville Kyrki,
• Abstract summary: Inverse reinforcement learning (IRL) is an imitation learning approach to learning reward functions from expert demonstrations. We propose a method that employs basis functions to form a candidate set of features. We demonstrate the approach's effectiveness by recovering reward functions that capture expert policies.
• Score: 7.278033100480175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inverse reinforcement learning (IRL) is an imitation learning approach to learning reward functions from expert demonstrations. Its use avoids the difficult and tedious procedure of manual reward specification while retaining the generalization power of reinforcement learning. In IRL, the reward is usually represented as a linear combination of features. In continuous state spaces, the state variables alone are not sufficiently rich to be used as features, but which features are good is not known in general. To address this issue, we propose a method that employs polynomial basis functions to form a candidate set of features, which are shown to allow the matching of statistical moments of state distributions. Feature selection is then performed for the candidates by leveraging the correlation between trajectory probabilities and feature expectations. We demonstrate the approach's effectiveness by recovering reward functions that capture expert policies across non-linear control tasks of increasing complexity. Code, data, and videos are available at https://sites.google.com/view/feature4irl.

Related papers

• Learning Causally Invariant Reward Functions from Diverse Demonstrations [6.351909403078771]
  Inverse reinforcement learning methods aim to retrieve the reward function of a Markov decision process based on a dataset of expert demonstrations. This adaptation often exhibits overfitting to the expert data set when a policy is trained on the obtained reward function under distribution shift of the environment dynamics. In this work, we explore a novel regularization approach for inverse reinforcement learning methods based on the causal invariance principle with the goal of improved reward function generalization.
  arXiv  Detail & Related papers  (2024-09-12T12:56:24Z)
• Offline Reinforcement Learning with Differentiable Function Approximation is Provably Efficient [65.08966446962845]
  offline reinforcement learning, which aims at optimizing decision-making strategies with historical data, has been extensively applied in real-life applications. We take a step by considering offline reinforcement learning with differentiable function class approximation (DFA) Most importantly, we show offline differentiable function approximation is provably efficient by analyzing the pessimistic fitted Q-learning algorithm.
  arXiv  Detail & Related papers  (2022-10-03T07:59:42Z)
• Invariance in Policy Optimisation and Partial Identifiability in Reward Learning [67.4640841144101]
  We characterise the partial identifiability of the reward function given popular reward learning data sources. We also analyse the impact of this partial identifiability for several downstream tasks, such as policy optimisation.
  arXiv  Detail & Related papers  (2022-03-14T20:19:15Z)
• Generative Adversarial Reward Learning for Generalized Behavior Tendency Inference [71.11416263370823]
  We propose a generative inverse reinforcement learning for user behavioral preference modelling. Our model can automatically learn the rewards from user's actions based on discriminative actor-critic network and Wasserstein GAN.
  arXiv  Detail & Related papers  (2021-05-03T13:14:25Z)
• Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification [133.20816939521941]
  In the standard Markov decision process formalism, users specify tasks by writing down a reward function. In many scenarios, the user is unable to describe the task in words or numbers, but can readily provide examples of what the world would look like if the task were solved. Motivated by this observation, we derive a control algorithm that aims to visit states that have a high probability of leading to successful outcomes, given only examples of successful outcome states.
  arXiv  Detail & Related papers  (2021-03-23T16:19:55Z)
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning [102.36450942613091]
  We propose an inverse reinforcement learning algorithm, called emphinverse temporal difference learning (ITD) We show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called $Psi Phi$-learning.
  arXiv  Detail & Related papers  (2021-02-24T21:12:09Z)
• Inverse Reinforcement Learning in the Continuous Setting with Formal Guarantees [31.122125783516726]
  Inverse Reinforcement Learning (IRL) is the problem of finding a reward function which describes observed/known expert behavior. We provide a new IRL algorithm for the continuous state space setting with unknown transition dynamics.
  arXiv  Detail & Related papers  (2021-02-16T03:17:23Z)
• Reward Propagation Using Graph Convolutional Networks [61.32891095232801]
  We propose a new framework for learning potential functions by leveraging ideas from graph representation learning. Our approach relies on Graph Convolutional Networks which we use as a key ingredient in combination with the probabilistic inference view of reinforcement learning.
  arXiv  Detail & Related papers  (2020-10-06T04:38:16Z)
• Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning [22.242379207077217]
  We show how to show the reward function's code to the RL agent so it can exploit the function's internal structure to learn optimal policies. First, we propose reward machines, a type of finite state machine that supports the specification of reward functions. We then describe different methodologies to exploit this structure to support learning, including automated reward shaping, task decomposition, and counterfactual reasoning with off-policy learning.
  arXiv  Detail & Related papers  (2020-10-06T00:10:16Z)
• Active Preference-Based Gaussian Process Regression for Reward Learning [42.697198807877925]
  One common approach is to learn reward functions from collected expert demonstrations. We present a preference-based learning approach, where as an alternative, the human feedback is only in the form of comparisons between trajectories. Our approach enables us to tackle both inflexibility and data-inefficiency problems within a preference-based learning framework.
  arXiv  Detail & Related papers  (2020-05-06T03:29:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.