On the Pros and Cons of Active Learning for Moral Preference Elicitation

• URL: http://arxiv.org/abs/2407.18889v1
• Date: Fri, 26 Jul 2024 17:40:52 GMT
• Title: On the Pros and Cons of Active Learning for Moral Preference Elicitation
• Abstract summary: We argue that the use of active learning for moral preference elicitation relies on certain assumptions about the underlying moral preferences. While these assumptions can be appropriate for preference elicitation in certain domains, prior research on moral psychology suggests they may not be valid for moral judgments.
• Score: 32.969662205265166
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Computational preference elicitation methods are tools used to learn people's preferences quantitatively in a given context. Recent works on preference elicitation advocate for active learning as an efficient method to iteratively construct queries (framed as comparisons between context-specific cases) that are likely to be most informative about an agent's underlying preferences. In this work, we argue that the use of active learning for moral preference elicitation relies on certain assumptions about the underlying moral preferences, which can be violated in practice. Specifically, we highlight the following common assumptions (a) preferences are stable over time and not sensitive to the sequence of presented queries, (b) the appropriate hypothesis class is chosen to model moral preferences, and (c) noise in the agent's responses is limited. While these assumptions can be appropriate for preference elicitation in certain domains, prior research on moral psychology suggests they may not be valid for moral judgments. Through a synthetic simulation of preferences that violate the above assumptions, we observe that active learning can have similar or worse performance than a basic random query selection method in certain settings. Yet, simulation results also demonstrate that active learning can still be viable if the degree of instability or noise is relatively small and when the agent's preferences can be approximately represented with the hypothesis class used for learning. Our study highlights the nuances associated with effective moral preference elicitation in practice and advocates for the cautious use of active learning as a methodology to learn moral preferences.

Related papers

• Learning the Preferences of a Learning Agent [11.820109078997504]
  inverse reinforcement learning aims to develop methods that allow for inferring preferences from observed behavior.<n>In this paper, we formalize the problem of learning the preferences of a learning agent.<n>We establish theoretical guarantees for various preference learning algorithms, or otherwise show that such guarantees are impossible.
  arXiv  Detail & Related papers  (2026-05-09T23:28:22Z)
• How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics [65.67654005892469]
  We show that proper instance-dependent sampling can yield stronger ranking guarantees, while skewed on-policy sampling can induce excessive concentration under structured preferences.<n>We then analyze iterative alignment dynamics in which the learned policy feeds back into future sampling and reference policies.<n>Our theoretical insights extend to Direct Preference Optimization, indicating the phenomena we captured are common to a broader class of preference-alignment methods.
  arXiv  Detail & Related papers  (2026-02-12T17:11:08Z)
• Wild Guesses and Mild Guesses in Active Concept Learning [0.0]
  We study a trade-off in a neuro-symbolic Bayesian learner whose hypotheses are proposed by a large language model (LLM)<n>We compare a Rational Active Learner that selects queries to maximize approximate expected information gain (EIG) and the human-like Positive Test Strategy (PTS)<n>Our results suggest that "confirmation bias" may not be a cognitive error, but rather a rational adaptation for maintaining tractable inference in the sparse, open-ended hypothesis spaces characteristic of human thought.
  arXiv  Detail & Related papers  (2026-02-06T16:04:44Z)
• Know When to Abstain: Optimal Selective Classification with Likelihood Ratios [10.317060648446828]
  We revisit the design of optimal selection functions through the lens of the Neyman-Pearson lemma.<n>We show that this perspective unifies the behavior of several post-hoc selection baselines, and also motivates new approaches to selective classification.<n>We evaluate our proposed methods across a range of vision and language tasks, including both supervised learning and vision-language models.
  arXiv  Detail & Related papers  (2025-05-21T01:26:21Z)
• Preference Optimization as Probabilistic Inference [21.95277469346728]
  We propose a method that can leverage unpaired preferred or dis-preferred examples, and works even when only one type of feedback is available. This flexibility allows us to apply it in scenarios with varying forms of feedback and models, including training generative language models.
  arXiv  Detail & Related papers  (2024-10-05T14:04:03Z)
• Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferences [12.775486996512434]
  Preference-Based reinforcement learning learns directly from the preferences of human teachers regarding agent behaviors. Existing PBRL methods often learn from explicit preferences, neglecting the possibility that teachers may choose equal preferences. We propose a novel PBRL method, Multi-Type Preference Learning (MTPL), which allows simultaneous learning from equal preferences while leveraging existing methods for learning from explicit preferences.
  arXiv  Detail & Related papers  (2024-09-11T13:43:49Z)
• Learning Linear Utility Functions From Pairwise Comparison Queries [35.01228510505625]
  We study learnability of linear utility functions from pairwise comparison queries. We show that in the passive learning setting, linear utilities are efficiently learnable with respect to the first objective. In this case, we show that even the second objective is efficiently learnable, and present algorithms for both the noise-free and noisy query response settings.
  arXiv  Detail & Related papers  (2024-05-04T08:43:45Z)
• A First Look at Selection Bias in Preference Elicitation for Recommendation [64.44255178199846]
  We study the effect of selection bias in preference elicitation on the resulting recommendations. A big hurdle is the lack of any publicly available dataset that has preference elicitation interactions. We propose a simulation of a topic-based preference elicitation process.
  arXiv  Detail & Related papers  (2024-05-01T14:56:56Z)
• Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data [102.16105233826917]
  Learning from preference labels plays a crucial role in fine-tuning large language models. There are several distinct approaches for preference fine-tuning, including supervised learning, on-policy reinforcement learning (RL), and contrastive learning.
  arXiv  Detail & Related papers  (2024-04-22T17:20:18Z)
• Reinforcement Learning Fine-tuning of Language Models is Biased Towards More Extractable Features [0.5937476291232802]
  We investigate whether principles governing inductive biases in the supervised fine-tuning of large language models also apply when the fine-tuning process uses reinforcement learning. We find statistically significant correlations which constitute strong evidence for these hypotheses.
  arXiv  Detail & Related papers  (2023-11-07T15:00:39Z)
• Generative Slate Recommendation with Reinforcement Learning [49.75985313698214]
  reinforcement learning algorithms can be used to optimize user engagement in recommender systems. However, RL approaches are intractable in the slate recommendation scenario. In that setting, an action corresponds to a slate that may contain any combination of items. In this work we propose to encode slates in a continuous, low-dimensional latent space learned by a variational auto-encoder. We are able to (i) relax assumptions required by previous work, and (ii) improve the quality of the action selection by modeling full slates.
  arXiv  Detail & Related papers  (2023-01-20T15:28:09Z)
• A Low Rank Promoting Prior for Unsupervised Contrastive Learning [108.91406719395417]
  We construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning. Our hypothesis explicitly requires that all the samples belonging to the same instance class lie on the same subspace with small dimension. Empirical evidences show that the proposed algorithm clearly surpasses the state-of-the-art approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2021-08-05T15:58:25Z)
• Exploration and preference satisfaction trade-off in reward-free learning [3.4998703934432682]
  We introduce a model-based Bayesian agent equipped with a preference learning mechanism using conjugate priors. We illustrate this in the OpenAI Gym FrozenLake and the 3D mini-world environments -- with and without volatility. Our experiments suggest that learnable (reward-free) preferences entail a trade-off between exploration and preference satisfaction.
  arXiv  Detail & Related papers  (2021-06-08T13:24:58Z)
• Online Active Model Selection for Pre-trained Classifiers [72.84853880948894]
  We design an online selective sampling approach that actively selects informative examples to label and outputs the best model with high probability at any round. Our algorithm can be used for online prediction tasks for both adversarial and streams.
  arXiv  Detail & Related papers  (2020-10-19T19:53:15Z)
• Let Me At Least Learn What You Really Like: Dealing With Noisy Humans When Learning Preferences [0.76146285961466]
  We propose a modification to uncertainty sampling which uses the expected output value to help speed up learning of preferences. We compare our approach with the uncertainty sampling baseline, as well as conduct an ablation study to test the validity of each component of our approach.
  arXiv  Detail & Related papers  (2020-02-15T00:36:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.