Assisted Robust Reward Design

• URL: http://arxiv.org/abs/2111.09884v1
• Date: Thu, 18 Nov 2021 18:59:33 GMT
• Title: Assisted Robust Reward Design
• Authors: Jerry Zhi-Yang He, Anca D. Dragan
• Abstract summary: In practice, reward design is an iterative process: the designer chooses a reward, eventually encounters an "edge-case" environment where the reward incentivizes the wrong behavior, revises the reward, and repeats. We propose that the robot not take the specified reward for granted, but rather have uncertainty about it, and account for the future design iterations as future evidence. We test this method in a simplified autonomous driving task and find that it more quickly improves the car's behavior in held-out environments by proposing environments that are "edge cases" for the current reward.
• Score: 33.55440481096258
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world robotic tasks require complex reward functions. When we define the problem the robot needs to solve, we pretend that a designer specifies this complex reward exactly, and it is set in stone from then on. In practice, however, reward design is an iterative process: the designer chooses a reward, eventually encounters an "edge-case" environment where the reward incentivizes the wrong behavior, revises the reward, and repeats. What would it mean to rethink robotics problems to formally account for this iterative nature of reward design? We propose that the robot not take the specified reward for granted, but rather have uncertainty about it, and account for the future design iterations as future evidence. We contribute an Assisted Reward Design method that speeds up the design process by anticipating and influencing this future evidence: rather than letting the designer eventually encounter failure cases and revise the reward then, the method actively exposes the designer to such environments during the development phase. We test this method in a simplified autonomous driving task and find that it more quickly improves the car's behavior in held-out environments by proposing environments that are "edge cases" for the current reward.

Related papers

• Sample-Efficient Curriculum Reinforcement Learning for Complex Reward Functions [5.78463306498655]
  Reinforcement learning (RL) shows promise in control problems, but its practical application is often hindered by the complexity arising from intricate reward functions with constraints. We propose a novel two-stage reward curriculum combined with a flexible replay buffer that adaptively samples experiences. Our approach first learns on a subset of rewards before transitioning to the full reward, allowing the agent to learn trade-offs between objectives and constraints.
  arXiv  Detail & Related papers  (2024-10-22T08:07:44Z)
• Synthesis of Reward Machines for Multi-Agent Equilibrium Design (Full Version) [2.2099217573031678]
  We study the problem of equilibrium design using dynamic incentive structures, known as reward machines. We show how reward machines can be used to represent dynamic incentives that allocate rewards in a manner that optimises the designer's goal. We present two variants of the problem: strong and weak. We demonstrate that both can be solved in time using a Turing machine equipped with an NP oracle.
  arXiv  Detail & Related papers  (2024-08-19T15:17:58Z)
• REBEL: A Regularization-Based Solution for Reward Overoptimization in Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and user intentions, values, or social norms can be catastrophic in the real world. Current methods to mitigate this misalignment work by learning reward functions from human preferences. We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft [88.80684763462384]
  This paper introduces an advanced learning system, named Auto MC-Reward, that leverages Large Language Models (LLMs) to automatically design dense reward functions. Experiments demonstrate a significant improvement in the success rate and learning efficiency of our agents in complex tasks in Minecraft.
  arXiv  Detail & Related papers  (2023-12-14T18:58:12Z)
• Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity [114.88145406445483]
  Reinforcement learning provides an automated framework for learning behaviors from high-level reward specifications. In practice the choice of reward function can be crucial for good results.
  arXiv  Detail & Related papers  (2022-10-18T04:21:25Z)
• Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards [46.068337522093096]
  We introduce the concept of motivation which captures the underlying goal of maximizing certain rewards. Our method performs better than the state-of-the-art methods in handling problems of delayed reward, exploration, and credit assignment.
  arXiv  Detail & Related papers  (2022-07-29T14:52:02Z)
• Programmatic Reward Design by Example [7.188571996124112]
  A misspecified or poorly designed reward can result in low sample efficiency and undesired behaviors. We propose the idea of textitprogrammatic reward design, i.e. using programs to specify the reward functions in reinforcement learning environments. A major contribution of this paper is a probabilistic framework that can infer the best candidate programmatic reward function from expert demonstrations.
  arXiv  Detail & Related papers  (2021-12-14T05:46:24Z)
• Reward (Mis)design for Autonomous Driving [89.2504219865973]
  We develop 8 simple sanity checks for identifying flaws in reward functions. The checks are applied to reward functions from past work on reinforcement learning for autonomous driving. We explore promising directions that may help future researchers design reward functions for AD.
  arXiv  Detail & Related papers  (2021-04-28T17:41:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.