Learning to Assist Humans without Inferring Rewards

• URL: http://arxiv.org/abs/2411.02623v2
• Date: Thu, 07 Nov 2024 09:25:28 GMT
• Title: Learning to Assist Humans without Inferring Rewards
• Authors: Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, Anca Dragan,
• Abstract summary: We build upon prior work that studies assistance through the lens of empowerment. An assistive agent aims to maximize the influence of the human's actions. We prove that these representations estimate a similar notion of empowerment to that studied by prior work.
• Score: 65.28156318196397
• License:
• Abstract: Assistive agents should make humans' lives easier. Classically, such assistance is studied through the lens of inverse reinforcement learning, where an assistive agent (e.g., a chatbot, a robot) infers a human's intention and then selects actions to help the human reach that goal. This approach requires inferring intentions, which can be difficult in high-dimensional settings. We build upon prior work that studies assistance through the lens of empowerment: an assistive agent aims to maximize the influence of the human's actions such that they exert a greater control over the environmental outcomes and can solve tasks in fewer steps. We lift the major limitation of prior work in this area--scalability to high-dimensional settings--with contrastive successor representations. We formally prove that these representations estimate a similar notion of empowerment to that studied by prior work and provide a ready-made mechanism for optimizing it. Empirically, our proposed method outperforms prior methods on synthetic benchmarks, and scales to Overcooked, a cooperative game setting. Theoretically, our work connects ideas from information theory, neuroscience, and reinforcement learning, and charts a path for representations to play a critical role in solving assistive problems.

Related papers

• Self-Explainable Affordance Learning with Embodied Caption [63.88435741872204]
  We introduce Self-Explainable Affordance learning (SEA) with embodied caption. SEA enables robots to articulate their intentions and bridge the gap between explainable vision-language caption and visual affordance learning. We propose a novel model to effectively combine affordance grounding with self-explanation in a simple but efficient manner.
  arXiv  Detail & Related papers  (2024-04-08T15:22:38Z)
• Optimising Human-AI Collaboration by Learning Convincing Explanations [62.81395661556852]
  We propose a method for a collaborative system that remains safe by having a human making decisions. Ardent enables efficient and effective decision-making by adapting to individual preferences for explanations.
  arXiv  Detail & Related papers  (2023-11-13T16:00:16Z)
• Robust Planning for Human-Robot Joint Tasks with Explicit Reasoning on Human Mental State [2.8246074016493457]
  We consider the human-aware task planning problem where a human-robot team is given a shared task with a known objective to achieve. Recent approaches tackle it by modeling it as a team of independent, rational agents, where the robot plans for both agents' (shared) tasks. We describe a novel approach to solve such problems, which models and uses execution-time observability conventions.
  arXiv  Detail & Related papers  (2022-10-17T09:21:00Z)
• Teaching Humans When To Defer to a Classifier via Examplars [9.851033166756274]
  We aim to ensure that human decision makers learn a valid mental model of the agent's strengths and weaknesses. We propose an exemplar-based teaching strategy where humans solve the task with the help of the agent. We present a novel parameterization of the human's mental model of the AI that applies a nearest neighbor rule in local regions.
  arXiv  Detail & Related papers  (2021-11-22T15:52:15Z)
• Contrastive Active Inference [12.361539023886161]
  We propose a contrastive objective for active inference that reduces the computational burden in learning the agent's generative model and planning future actions. Our method performs notably better than likelihood-based active inference in image-based tasks, while also being computationally cheaper and easier to train.
  arXiv  Detail & Related papers  (2021-10-19T16:20:49Z)
• Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks [60.380501589764144]
  A longstanding goal of artificial intelligence is to create artificial agents capable of learning to perform tasks that require sequential decision making. While it is the artificial agent that learns and acts, it is still up to humans to specify the particular task to be performed. This survey provides a high-level overview of five recent machine learning frameworks that primarily rely on human guidance.
  arXiv  Detail & Related papers  (2021-07-13T03:11:04Z)
• Learning Human Rewards by Inferring Their Latent Intelligence Levels in Multi-Agent Games: A Theory-of-Mind Approach with Application to Driving Data [18.750834997334664]
  We argue that humans are bounded rational and have different intelligence levels when reasoning about others' decision-making process. We propose a new multi-agent Inverse Reinforcement Learning framework that reasons about humans' latent intelligence levels during learning.
  arXiv  Detail & Related papers  (2021-03-07T07:48:31Z)
• AvE: Assistance via Empowerment [77.08882807208461]
  We propose a new paradigm for assistance by instead increasing the human's ability to control their environment. This task-agnostic objective preserves the person's autonomy and ability to achieve any eventual state.
  arXiv  Detail & Related papers  (2020-06-26T04:40:11Z)
• Automatic Curriculum Learning through Value Disagreement [95.19299356298876]
  Continually solving new, unsolved tasks is the key to learning diverse behaviors. In the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. We propose setting up an automatic curriculum for goals that the agent needs to solve. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T03:58:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.