The Expertise Problem: Learning from Specialized Feedback

• URL: http://arxiv.org/abs/2211.06519v1
• Date: Sat, 12 Nov 2022 00:07:35 GMT
• Title: The Expertise Problem: Learning from Specialized Feedback
• Authors: Oliver Daniels-Koch, Rachel Freedman
• Abstract summary: Reinforcement learning from human feedback (RLHF) is a powerful technique for training agents to perform difficult-to-specify tasks. Levels of expertise vary across teachers, and a given teacher may have differing levels of expertise for different components of a task. Existing RLHF algorithms assume that all evaluations come from the same distribution, obscuring this inter- and intra-human variance.
• Score: 7.858296711223292
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning from human feedback (RLHF) is a powerful technique for training agents to perform difficult-to-specify tasks. However, human feedback can be noisy, particularly when human teachers lack relevant knowledge or experience. Levels of expertise vary across teachers, and a given teacher may have differing levels of expertise for different components of a task. RLHF algorithms that learn from multiple teachers therefore face an expertise problem: the reliability of a given piece of feedback depends both on the teacher that it comes from and how specialized that teacher is on relevant components of the task. Existing state-of-the-art RLHF algorithms assume that all evaluations come from the same distribution, obscuring this inter- and intra-human variance, and preventing them from accounting for or taking advantage of variations in expertise. We formalize this problem, implement it as an extension of an existing RLHF benchmark, evaluate the performance of a state-of-the-art RLHF algorithm, and explore techniques to improve query and teacher selection. Our key contribution is to demonstrate and characterize the expertise problem, and to provide an open-source implementation for testing future solutions.

Related papers

• Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition [24.293448609592147]
  Multi-Teacher Knowledge Distillation (KD) transfers diverse knowledge from a teacher pool to a student network. This paper proposes Multi-Teacher Knowledge Distillation with Reinforcement Learning (MTKD-RL) to optimize multi-teacher weights.
  arXiv  Detail & Related papers  (2025-02-22T09:31:24Z)
• What Makes An Expert? Reviewing How ML Researchers Define "Expert" [4.6346970187885885]
  We review 112 academic publications that explicitly reference 'expert' and 'expertise' We find that expertise is often undefined and forms of knowledge outside of formal education are rarely sought. We discuss the ways experts are engaged in ML development in relation to deskilling, the social construction of expertise, and implications for responsible AI development.
  arXiv  Detail & Related papers  (2024-10-31T19:51:28Z)
• CANDERE-COACH: Reinforcement Learning from Noisy Feedback [12.232688822099325]
  The CANDERE-COACH algorithm is capable of learning from noisy feedback by a nonoptimal teacher. We propose a noise-filtering mechanism to de-noise online feedback data, thereby enabling the RL agent to successfully learn with up to 40% of the teacher feedback being incorrect.
  arXiv  Detail & Related papers  (2024-09-23T20:14:12Z)
• RILe: Reinforced Imitation Learning [60.63173816209543]
  RILe is a novel trainer-student system that learns a dynamic reward function based on the student's performance and alignment with expert demonstrations. RILe enables better performance in complex settings where traditional methods falter, outperforming existing methods by 2x in complex simulated robot-locomotion tasks.
  arXiv  Detail & Related papers  (2024-06-12T17:56:31Z)
• RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs [49.386699863989335]
  Training large language models (LLMs) to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals.
  arXiv  Detail & Related papers  (2024-04-12T15:54:15Z)
• The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education [3.967610895056427]
  This paper presents the first study that leverages Natural Language Processing (NLP) techniques to assess multiple high-inference instructional practices. We confront two challenges inherent in NLP-based instructional analysis, including noisy and long input data and highly skewed distributions of human ratings.
  arXiv  Detail & Related papers  (2024-04-03T04:15:29Z)
• Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts [58.220879689376744]
  Reinforcement learning (RL) is a powerful approach for acquiring a good-performing policy. We propose textbfDiverse textbfSkill textbfLearning (Di-SkilL) for learning diverse skills. We show on challenging robot simulation tasks that Di-SkilL can learn diverse and performant skills.
  arXiv  Detail & Related papers  (2024-03-11T17:49:18Z)
• Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts [81.37287967870589]
  We propose to harness a diverse set of specialized teachers, instead of a single generalist one, that collectively supervises the strong student. Our approach resembles the classical hierarchical mixture of experts, with two components tailored for co-supervision. We validate the proposed method through visual recognition tasks on the OpenAI weak-to-strong benchmark and additional multi-domain datasets.
  arXiv  Detail & Related papers  (2024-02-23T18:56:11Z)
• Active teacher selection for reinforcement learning from human feedback [14.009227941725783]
  Reinforcement learning from human feedback (RLHF) enables machine learning systems to learn objectives from human feedback. We propose the Hidden Utility Bandit framework to model differences in teacher rationality, expertise, and costliness. We develop a variety of solution algorithms and apply them to two real-world domains: paper recommendation systems and COVID-19 vaccine testing.
  arXiv  Detail & Related papers  (2023-10-23T18:54:43Z)
• Rethinking Learning Dynamics in RL using Adversarial Networks [79.56118674435844]
  We present a learning mechanism for reinforcement learning of closely related skills parameterized via a skill embedding space. The main contribution of our work is to formulate an adversarial training regime for reinforcement learning with the help of entropy-regularized policy gradient formulation.
  arXiv  Detail & Related papers  (2022-01-27T19:51:09Z)
• ArT: All-round Thinker for Unsupervised Commonsense Question-Answering [54.068032948300655]
  We propose an approach of All-round Thinker (ArT) by fully taking association during knowledge generating. We evaluate it on three commonsense QA benchmarks: COPA, SocialIQA and SCT.
  arXiv  Detail & Related papers  (2021-12-26T18:06:44Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• Neural Multi-Task Learning for Teacher Question Detection in Online Classrooms [50.19997675066203]
  We build an end-to-end neural framework that automatically detects questions from teachers' audio recordings. By incorporating multi-task learning techniques, we are able to strengthen the understanding of semantic relations among different types of questions.
  arXiv  Detail & Related papers  (2020-05-16T02:17:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.