Exploration and preference satisfaction trade-off in reward-free learning

• URL: http://arxiv.org/abs/2106.04316v1
• Date: Tue, 8 Jun 2021 13:24:58 GMT
• Title: Exploration and preference satisfaction trade-off in reward-free learning
• Abstract summary: We introduce a model-based Bayesian agent equipped with a preference learning mechanism using conjugate priors. We illustrate this in the OpenAI Gym FrozenLake and the 3D mini-world environments -- with and without volatility. Our experiments suggest that learnable (reward-free) preferences entail a trade-off between exploration and preference satisfaction.
• Score: 3.4998703934432682
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Biological agents have meaningful interactions with their environment despite the absence of a reward signal. In such instances, the agent can learn preferred modes of behaviour that lead to predictable states -- necessary for survival. In this paper, we pursue the notion that this learnt behaviour can be a consequence of reward-free preference learning that ensures an appropriate trade-off between exploration and preference satisfaction. For this, we introduce a model-based Bayesian agent equipped with a preference learning mechanism (pepper) using conjugate priors. These conjugate priors are used to augment the expected free energy planner for learning preferences over states (or outcomes) across time. Importantly, our approach enables the agent to learn preferences that encourage adaptive behaviour at test time. We illustrate this in the OpenAI Gym FrozenLake and the 3D mini-world environments -- with and without volatility. Given a constant environment, these agents learn confident (i.e., precise) preferences and act to satisfy them. Conversely, in a volatile setting, perpetual preference uncertainty maintains exploratory behaviour. Our experiments suggest that learnable (reward-free) preferences entail a trade-off between exploration and preference satisfaction. Pepper offers a straightforward framework suitable for designing adaptive agents when reward functions cannot be predefined as in real environments.

Related papers

• Learning the Preferences of a Learning Agent [11.820109078997504]
  inverse reinforcement learning aims to develop methods that allow for inferring preferences from observed behavior.<n>In this paper, we formalize the problem of learning the preferences of a learning agent.<n>We establish theoretical guarantees for various preference learning algorithms, or otherwise show that such guarantees are impossible.
  arXiv  Detail & Related papers  (2026-05-09T23:28:22Z)
• Environment Agnostic Goal-Conditioning, A Study of Reward-Free Autonomous Learning [0.0]
  We show that an agent can learn to solve tasks by selecting its own goals in an environment-agnostic way.<n>Our method is independent of the underlying off-policy learning algorithm.
  arXiv  Detail & Related papers  (2025-11-06T17:51:11Z)
• On the Pros and Cons of Active Learning for Moral Preference Elicitation [32.969662205265166]
  We argue that the use of active learning for moral preference elicitation relies on certain assumptions about the underlying moral preferences. While these assumptions can be appropriate for preference elicitation in certain domains, prior research on moral psychology suggests they may not be valid for moral judgments.
  arXiv  Detail & Related papers  (2024-07-26T17:40:52Z)
• Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input [17.131441665935128]
  We study how to extract fine-grained data regarding why an example is preferred that is useful for learning more accurate reward models. Our findings suggest that incorporating pragmatic feature preferences is a promising approach for more efficient user-aligned reward learning.
  arXiv  Detail & Related papers  (2024-05-23T16:36:16Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• What Should I Know? Using Meta-gradient Descent for Predictive Feature Discovery in a Single Stream of Experience [63.75363908696257]
  computational reinforcement learning seeks to construct an agent's perception of the world through predictions of future sensations. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. We introduce a meta-gradient descent process by which an agent learns what predictions to make, 2) the estimates for its chosen predictions, and 3) how to use those estimates to generate policies that maximize future reward.
  arXiv  Detail & Related papers  (2022-06-13T21:31:06Z)
• Models of human preference for learning reward functions [80.39289349661364]
  We learn the reward function from human-generated preferences between pairs of trajectory segments. We find this assumption to be flawed and propose modeling human preferences as informed by each segment's regret. Our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned.
  arXiv  Detail & Related papers  (2022-06-05T17:58:02Z)
• Causal Confusion and Reward Misidentification in Preference-Based Reward Learning [33.944367978407904]
  We study causal confusion and reward misidentification when learning from preferences. We find that the presence of non-causal distractor features, noise in the stated preferences, and partial state observability can all exacerbate reward misidentification.
  arXiv  Detail & Related papers  (2022-04-13T18:41:41Z)
• Safer Autonomous Driving in a Stochastic, Partially-Observable Environment by Hierarchical Contingency Planning [10.971411555103574]
  An intelligent agent should be prepared to anticipate a change in its belief of the environment state. This is especially the case for autonomous vehicles (AVs) navigating real-world situations where safety is paramount. We show that our approach results in robust, safe behaviour in a partially observable, safe environment, generalizing well over environment not seen during training.
  arXiv  Detail & Related papers  (2022-04-13T16:47:00Z)
• Ecological Reinforcement Learning [76.9893572776141]
  We study the kinds of environment properties that can make learning under such conditions easier. understanding how properties of the environment impact the performance of reinforcement learning agents can help us to structure our tasks in ways that make learning tractable.
  arXiv  Detail & Related papers  (2020-06-22T17:55:03Z)
• Safe Reinforcement Learning via Curriculum Induction [94.67835258431202]
  In safety-critical applications, autonomous agents may need to learn in an environment where mistakes can be very costly. Existing safe reinforcement learning methods make an agent rely on priors that let it avoid dangerous situations. This paper presents an alternative approach inspired by human teaching, where an agent learns under the supervision of an automatic instructor.
  arXiv  Detail & Related papers  (2020-06-22T10:48:17Z)
• Curiosity Killed or Incapacitated the Cat and the Asymptotically Optimal Agent [21.548271801592907]
  Reinforcement learners are agents that learn to pick actions that lead to high reward. We show that if an agent is guaranteed to be "asymptotically optimal" in any environment, then subject to an assumption about the true environment, this agent will be either "destroyed" or "incapacitated" We present an agent, Mentee, with the modest guarantee of approaching the performance of a mentor, doing safe exploration instead of reckless exploration.
  arXiv  Detail & Related papers  (2020-06-05T10:42:29Z)
• Intrinsic Motivation for Encouraging Synergistic Behavior [55.10275467562764]
  We study the role of intrinsic motivation as an exploration bias for reinforcement learning in sparse-reward synergistic tasks. Our key idea is that a good guiding principle for intrinsic motivation in synergistic tasks is to take actions which affect the world in ways that would not be achieved if the agents were acting on their own.
  arXiv  Detail & Related papers  (2020-02-12T19:34:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.