Learning Shared Safety Constraints from Multi-task Demonstrations
- URL: http://arxiv.org/abs/2309.00711v1
- Date: Fri, 1 Sep 2023 19:37:36 GMT
- Title: Learning Shared Safety Constraints from Multi-task Demonstrations
- Authors: Konwoo Kim, Gokul Swamy, Zuxin Liu, Ding Zhao, Sanjiban Choudhury,
Zhiwei Steven Wu
- Abstract summary: We show how to learn constraints from expert demonstrations of safe task completion.
We learn constraints that forbid highly rewarding behavior that the expert could have taken but chose not to.
We validate our method with simulation experiments on high-dimensional continuous control tasks.
- Score: 53.116648461888936
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Regardless of the particular task we want them to perform in an environment,
there are often shared safety constraints we want our agents to respect. For
example, regardless of whether it is making a sandwich or clearing the table, a
kitchen robot should not break a plate. Manually specifying such a constraint
can be both time-consuming and error-prone. We show how to learn constraints
from expert demonstrations of safe task completion by extending inverse
reinforcement learning (IRL) techniques to the space of constraints.
Intuitively, we learn constraints that forbid highly rewarding behavior that
the expert could have taken but chose not to. Unfortunately, the constraint
learning problem is rather ill-posed and typically leads to overly conservative
constraints that forbid all behavior that the expert did not take. We counter
this by leveraging diverse demonstrations that naturally occur in multi-task
settings to learn a tighter set of constraints. We validate our method with
simulation experiments on high-dimensional continuous control tasks.
Related papers
- DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systems [13.93024489228903]
predefined constraint functions to ensure safety in complex real-world tasks, such as autonomous driving.
Recent research highlights the potential of leveraging pre-acquired task-agnostic knowledge to enhance both safety and sample efficiency in related tasks.
We propose a novel method to learn shared constraint distributions across multiple tasks.
Our approach identifies the shared constraints through imitation learning and then adapts to new tasks by adjusting risk levels within these learned distributions.
arXiv Detail & Related papers (2025-01-30T01:56:07Z) - From Instructions to Constraints: Language Model Alignment with
Automatic Constraint Verification [70.08146540745877]
We investigate common constraints in NLP tasks, categorize them into three classes based on the types of their arguments.
We propose a unified framework, ACT (Aligning to ConsTraints), to automatically produce supervision signals for user alignment with constraints.
arXiv Detail & Related papers (2024-03-10T22:14:54Z) - Dynamic Interval Restrictions on Action Spaces in Deep Reinforcement
Learning for Obstacle Avoidance [0.0]
In this thesis, we consider the problem of interval restrictions as they occur in pathfinding with dynamic obstacles.
Recent research learns with strong assumptions on the number of intervals, is limited to convex subsets.
We propose two approaches that are independent of the state of the environment by extending parameterized reinforcement learning and ConstraintNet to handle an arbitrary number of intervals.
arXiv Detail & Related papers (2023-06-13T09:13:13Z) - Resilient Constrained Learning [94.27081585149836]
This paper presents a constrained learning approach that adapts the requirements while simultaneously solving the learning task.
We call this approach resilient constrained learning after the term used to describe ecological systems that adapt to disruptions by modifying their operation.
arXiv Detail & Related papers (2023-06-04T18:14:18Z) - Controlled Text Generation with Natural Language Instructions [74.88938055638636]
InstructCTG is a controlled text generation framework that incorporates different constraints.
We first extract the underlying constraints of natural texts through a combination of off-the-shelf NLP tools and simple verbalizes.
By prepending natural language descriptions of the constraints and a few demonstrations, we fine-tune a pre-trained language model to incorporate various types of constraints.
arXiv Detail & Related papers (2023-04-27T15:56:34Z) - Constrained Reinforcement Learning for Dexterous Manipulation [0.6193838300896449]
We investigate the effects of adding position-based constraints to a 24-DOF robot hand learning to perform object relocation.
We find that a simple geometric constraint can ensure the robot learns to move towards the object sooner than without constraints.
These findings shed light on how simple constraints can help robots achieve sensible and safe behavior quickly and ease concerns surrounding hardware deployment.
arXiv Detail & Related papers (2023-01-24T00:31:28Z) - Learning Soft Constraints From Constrained Expert Demonstrations [16.442694252601452]
Inverse reinforcement learning (IRL) methods assume that the expert data is generated by an agent optimizing some reward function.
We consider the setting where the reward function is given, and the constraints are unknown, and propose a method that is able to recover these constraints satisfactorily from the expert data.
We demonstrate our approach on synthetic environments, robotics environments and real world highway driving scenarios.
arXiv Detail & Related papers (2022-06-02T21:45:31Z) - Inverse Constrained Reinforcement Learning [12.669649178762718]
In this work, we consider the problem of learning constraints from demonstrations of a constraint-abiding agent's behavior.
We show that our framework can successfully learn the most likely constraints that the agent respects.
These learned constraints are textittransferable to new agents that may have different morphologies and/or reward functions.
arXiv Detail & Related papers (2020-11-19T17:56:33Z) - Constrained episodic reinforcement learning in concave-convex and
knapsack settings [81.08055425644037]
We provide a modular analysis with strong theoretical guarantees for settings with concave rewards and convex constraints.
Our experiments demonstrate that the proposed algorithm significantly outperforms these approaches in existing constrained episodic environments.
arXiv Detail & Related papers (2020-06-09T05:02:44Z) - Weakly-Supervised Reinforcement Learning for Controllable Behavior [126.04932929741538]
Reinforcement learning (RL) is a powerful framework for learning to take actions to solve tasks.
In many settings, an agent must winnow down the inconceivably large space of all possible tasks to the single task that it is currently being asked to solve.
We introduce a framework for using weak supervision to automatically disentangle this semantically meaningful subspace of tasks from the enormous space of nonsensical "chaff" tasks.
arXiv Detail & Related papers (2020-04-06T17:50:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.