Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoning

• URL: http://arxiv.org/abs/2311.08385v4
• Date: Fri, 01 Nov 2024 10:28:12 GMT
• Title: Aligning Large Language Models with Human Opinions through Persona Selection and Value--Belief--Norm Reasoning
• Authors: Do Xuan Long, Kenji Kawaguchi, Min-Yen Kan, Nancy F. Chen,
• Abstract summary: Chain-of-Opinion (COO) is a simple four-step solution modeling which and how to reason with personae. COO distinguishes between explicit personae (demographics and ideology) and implicit personae (historical opinions) COO efficiently achieves new state-of-the-art opinion prediction via prompting with only 5 inference calls, improving prior techniques by up to 4%.
• Score: 67.33899440998175
• License:
• Abstract: Reasoning and predicting human opinions with large language models (LLMs) is essential yet challenging. Current methods employ role-playing with personae but face two major issues: LLMs are sensitive to even a single irrelevant persona, skewing predictions by up to 30%, and LLMs fail to reason strategically over personae. We propose Chain-of-Opinion (COO), a simple four-step solution modeling which and how to reason with personae, inspired by the Value--Belief--Norm (VBN) theory. COO differentiates between explicit personae (demographics and ideology) and implicit personae (historical opinions), involves: (1) filtering irrelevant attributes from explicit personae, (2) ranking implicit personae into a preferential list for selecting top-k, (3) applying novel VBN reasoning to extract user environmental and personal value, belief, and norm variables for accurate and reliable predictions, and (4) iterating VBN reasoning with progressively larger lists of implicit personae to handle potential persona insufficiency. COO efficiently achieves new state-of-the-art opinion prediction via prompting with only 5 inference calls, improving prior techniques by up to 4%. Notably, fine-tuning LMs with COO data results in significantly better opinion-aligned models, by up to 23%.

Related papers

• Large Language Models Assume People are More Rational than We Really are [10.857040292234984]
  In order for AI to communicate effectively with people, they must understand how we make decisions. Previous empirical evidence seems to suggest that these implicit models are accurate. We find that this is actually not the case when both simulating and predicting people's choices.
  arXiv  Detail & Related papers  (2024-06-24T18:15:27Z)
• Evaluating Large Language Model Biases in Persona-Steered Generation [26.92498998306013]
  We show that large language models (LLMs) are 9.7% less steerable towards incongruous personas than congruous ones. Models that are fine-tuned with Reinforcement Learning from Human Feedback (RLHF) are more steerable, especially towards stances associated with political liberals and women.
  arXiv  Detail & Related papers  (2024-05-30T17:06:03Z)
• Large Language Models Show Human-like Social Desirability Biases in Survey Responses [12.767606361552684]
  We show that Large Language Models (LLMs) skew their scores towards the desirable ends of trait dimensions when personality evaluation is inferred. This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. reverse-coding all the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias.
  arXiv  Detail & Related papers  (2024-05-09T19:02:53Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)
• Quantifying the Persona Effect in LLM Simulations [25.367927300697424]
  Large language models (LLMs) have shown remarkable promise in simulating human language and behavior. This study investigates how integrating persona variables-demographic, social, and behavioral factors-impacts LLMs' ability to simulate diverse perspectives. We find that persona variables account for 10% variance in annotations in existing subjective NLP datasets.
  arXiv  Detail & Related papers  (2024-02-16T16:35:35Z)
• Fine-tuning Language Models for Factuality [96.5203774943198]
  Large pre-trained language models (LLMs) have led to their widespread use, sometimes even as a replacement for traditional search engines. Yet language models are prone to making convincing but factually inaccurate claims, often referred to as 'hallucinations' In this work, we fine-tune language models to be more factual, without human labeling.
  arXiv  Detail & Related papers  (2023-11-14T18:59:15Z)
• Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties [68.66719970507273]
  Value pluralism is the view that multiple correct values may be held in tension with one another. As statistical learners, AI systems fit to averages by default, washing out potentially irreducible value conflicts. We introduce ValuePrism, a large-scale dataset of 218k values, rights, and duties connected to 31k human-written situations.
  arXiv  Detail & Related papers  (2023-09-02T01:24:59Z)
• Training Language Models with Language Feedback at Scale [50.70091340506957]
  We introduce learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback.
  arXiv  Detail & Related papers  (2023-03-28T17:04:15Z)
• Fine-tuning language models to find agreement among humans with diverse preferences [7.702628192754256]
  Recent work in large language modeling (LLMs) has used fine-tuning to align outputs with the preferences of a prototypical user. Here, we consider how might a machine help people with diverse views find agreement? We fine-tune a 70 billion parameter LLM to generate statements that maximize the expected approval for a group of people with potentially diverse opinions. We find that when we silently constructed consensus statements from only a subset of group members, those who were excluded were more likely to dissent.
  arXiv  Detail & Related papers  (2022-11-28T02:24:14Z)
• Dialogue Response Ranking Training with Large-Scale Human Feedback Data [52.12342165926226]
  We leverage social media feedback data to build a large-scale training dataset for feedback prediction. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data. Our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback.
  arXiv  Detail & Related papers  (2020-09-15T10:50:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.