Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaboration

• URL: http://arxiv.org/abs/2211.01602v1
• Date: Thu, 3 Nov 2022 06:10:22 GMT
• Title: Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaboration
• Authors: Mesut Yang, Micah Carroll, Anca Dragan
• Abstract summary: We show that using optimal behavior as a prior for human models makes these models vastly more data-efficient. We also show that using these improved human models often leads to better human-AI collaboration performance.
• Score: 0.5524804393257919
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents designed to collaborate with people benefit from models that enable them to anticipate human behavior. However, realistic models tend to require vast amounts of human data, which is often hard to collect. A good prior or initialization could make for more data-efficient training, but what makes for a good prior on human behavior? Our work leverages a very simple assumption: people generally act closer to optimal than to random chance. We show that using optimal behavior as a prior for human models makes these models vastly more data-efficient and able to generalize to new environments. Our intuition is that such a prior enables the training to focus one's precious real-world data on capturing the subtle nuances of human suboptimality, instead of on the basics of how to do the task in the first place. We also show that using these improved human models often leads to better human-AI collaboration performance compared to using models based on real human data alone.

Related papers

• Learning to Cooperate with Humans using Generative Agents [40.605931138995714]
  Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL) We show emphlearning a generative model of human partners can effectively address this issue. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents.
  arXiv  Detail & Related papers  (2024-11-21T08:36:17Z)
• Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF [79.98542868281471]
  Reinforcement Learning from Human Feedback (RLHF) is a technique that aligns language models closely with human-centric values. It is observed that the performance of the reward model degrades after one epoch of training, and optimizing too much against the learned reward model eventually hinders the true objective. This paper delves into these issues, leveraging the theoretical insights to design improved reward learning algorithm termed 'Iterative Data Smoothing' (IDS)
  arXiv  Detail & Related papers  (2024-01-29T17:43:42Z)
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
  Fine-tuning language models(LMs) on human-generated data remains a prevalent practice. We investigate whether we can go beyond human data on tasks where we have access to scalar feedback. We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
  arXiv  Detail & Related papers  (2023-12-11T18:17:43Z)
• Learning Human Action Recognition Representations Without Real Humans [66.61527869763819]
  We present a benchmark that leverages real-world videos with humans removed and synthetic data containing virtual humans to pre-train a model. We then evaluate the transferability of the representation learned on this data to a diverse set of downstream action recognition benchmarks. Our approach outperforms previous baselines by up to 5%.
  arXiv  Detail & Related papers  (2023-11-10T18:38:14Z)
• Learning Defect Prediction from Unrealistic Data [57.53586547895278]
  Pretrained models of code have become popular choices for code understanding and generation tasks. Such models tend to be large and require commensurate volumes of training data. It has become popular to train models with far larger but less realistic datasets, such as functions with artificially injected bugs. Models trained on such data tend to only perform well on similar data, while underperforming on real world programs.
  arXiv  Detail & Related papers  (2023-11-02T01:51:43Z)
• Are Sample-Efficient NLP Models More Robust? [90.54786862811183]
  We investigate the relationship between sample efficiency (amount of data needed to reach a given ID accuracy) and robustness (how models fare on OOD evaluation) We find that higher sample efficiency is only correlated with better average OOD robustness on some modeling interventions and tasks, but not others. These results suggest that general-purpose methods for improving sample efficiency are unlikely to yield universal OOD robustness improvements, since such improvements are highly dataset- and task-dependent.
  arXiv  Detail & Related papers  (2022-10-12T17:54:59Z)
• Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning [13.64577704565643]
  We argue that these models are too simplistic and that RL researchers need to develop more realistic human models to design and evaluate their algorithms. This paper calls for research from different disciplines to address key questions about how humans provide feedback to AIs and how we can build more robust human-in-the-loop RL systems.
  arXiv  Detail & Related papers  (2022-06-27T13:58:51Z)
• The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models [5.736353542430439]
  We introduce the Boltzmann policy distribution (BPD), which serves as a prior over human policies. BPD adapts via Bayesian inference to capture systematic deviations by observing human actions during a single episode. We show that the BPD enables prediction of human behavior and human-AI collaboration equally as well as imitation learning-based human models.
  arXiv  Detail & Related papers  (2022-04-22T15:26:25Z)
• Investigations of Performance and Bias in Human-AI Teamwork in Hiring [30.046502708053097]
  In AI-assisted decision-making, effective hybrid teamwork (human-AI) is not solely dependent on AI performance alone. We investigate how both a model's predictive performance and bias may transfer to humans in a recommendation-aided decision task.
  arXiv  Detail & Related papers  (2022-02-21T17:58:07Z)
• Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback [82.96694147237113]
  We present Skill Preferences, an algorithm that learns a model over human preferences and uses it to extract human-aligned skills from offline data. We show that SkiP enables a simulated kitchen robot to solve complex multi-step manipulation tasks.
  arXiv  Detail & Related papers  (2021-08-11T18:04:08Z)
• Humans learn too: Better Human-AI Interaction using Optimized Human Inputs [2.5991265608180396]
  Humans rely more and more on systems with AI components. The AI community typically treats human inputs as a given and optimize AI models only. In this work, human inputs are optimized for better interaction with an AI model while keeping the model fixed.
  arXiv  Detail & Related papers  (2020-09-19T16:30:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.