Large Language Models as Superpositions of Cultural Perspectives
- URL: http://arxiv.org/abs/2307.07870v3
- Date: Tue, 7 Nov 2023 16:28:33 GMT
- Title: Large Language Models as Superpositions of Cultural Perspectives
- Authors: Grgur Kova\v{c}, Masataka Sawayama, R\'emy Portelas, C\'edric Colas,
Peter Ford Dominey, Pierre-Yves Oudeyer
- Abstract summary: Large Language Models (LLMs) are often misleadingly recognized as having a personality or a set of values.
We argue that an LLM can be seen as a superposition of perspectives with different values and personality traits.
- Score: 25.114678091641935
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large Language Models (LLMs) are often misleadingly recognized as having a
personality or a set of values. We argue that an LLM can be seen as a
superposition of perspectives with different values and personality traits.
LLMs exhibit context-dependent values and personality traits that change based
on the induced perspective (as opposed to humans, who tend to have more
coherent values and personality traits across contexts). We introduce the
concept of perspective controllability, which refers to a model's affordance to
adopt various perspectives with differing values and personality traits. In our
experiments, we use questionnaires from psychology (PVQ, VSM, IPIP) to study
how exhibited values and personality traits change based on different
perspectives. Through qualitative experiments, we show that LLMs express
different values when those are (implicitly or explicitly) implied in the
prompt, and that LLMs express different values even when those are not
obviously implied (demonstrating their context-dependent nature). We then
conduct quantitative experiments to study the controllability of different
models (GPT-4, GPT-3.5, OpenAssistant, StableVicuna, StableLM), the
effectiveness of various methods for inducing perspectives, and the smoothness
of the models' drivability. We conclude by examining the broader implications
of our work and outline a variety of associated scientific questions. The
project website is available at
https://sites.google.com/view/llm-superpositions .
Related papers
- One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity [2.5975241792179378]
Researchers have proposed using large language models (LLMs) as replacements for humans in behavioral research.
It is debated whether post-training alignment (RLHF or RLAIF) affects models' internal diversity.
We use a new way of measuring the conceptual diversity of synthetically-generated LLM "populations" by relating the internal variability of simulated individuals to the population-level variability.
arXiv Detail & Related papers (2024-11-07T04:38:58Z) - Large Language Models Reflect the Ideology of their Creators [73.25935570218375]
Large language models (LLMs) are trained on vast amounts of data to generate natural language.
We uncover notable diversity in the ideological stance exhibited across different LLMs and languages.
arXiv Detail & Related papers (2024-10-24T04:02:30Z) - Bias in the Mirror: Are LLMs opinions robust to their own adversarial attacks ? [22.0383367888756]
Large language models (LLMs) inherit biases from their training data and alignment processes, influencing their responses in subtle ways.
We introduce a novel approach where two instances of an LLM engage in self-debate, arguing opposing viewpoints to persuade a neutral version of the model.
We evaluate how firmly biases hold and whether models are susceptible to reinforcing misinformation or shifting to harmful viewpoints.
arXiv Detail & Related papers (2024-10-17T13:06:02Z) - VHELM: A Holistic Evaluation of Vision Language Models [75.88987277686914]
We present the Holistic Evaluation of Vision Language Models (VHELM)
VHELM aggregates various datasets to cover one or more of the 9 aspects: visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety.
Our framework is designed to be lightweight and automatic so that evaluation runs are cheap and fast.
arXiv Detail & Related papers (2024-10-09T17:46:34Z) - Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models [61.45529177682614]
We challenge the prevailing constrained evaluation paradigm for values and opinions in large language models.
We show that models give substantively different answers when not forced.
We distill these findings into recommendations and open challenges in evaluating values and opinions in LLMs.
arXiv Detail & Related papers (2024-02-26T18:00:49Z) - Eliciting Personality Traits in Large Language Models [0.0]
Large Language Models (LLMs) are increasingly being utilized by both candidates and employers in the recruitment context.
This study seeks to obtain a better understanding of such models by examining their output variations based on different input prompts.
arXiv Detail & Related papers (2024-02-13T10:09:00Z) - Illuminating the Black Box: A Psychometric Investigation into the
Multifaceted Nature of Large Language Models [3.692410936160711]
This study explores the idea of AI Personality or AInality suggesting that Large Language Models (LLMs) exhibit patterns similar to human personalities.
Using projective tests, we uncover hidden aspects of LLM personalities that are not easily accessible through direct questioning.
Our machine learning analysis revealed that LLMs exhibit distinct AInality traits and manifest diverse personality types, demonstrating dynamic shifts in response to external instructions.
arXiv Detail & Related papers (2023-12-21T04:57:21Z) - How Far Can We Extract Diverse Perspectives from Large Language Models? [16.16678226707335]
We show that large language models (LLMs) can generate diverse perspectives on subjective topics.
We propose a criteria-based prompting technique to ground diverse opinions.
Our methods, applied to various tasks, show that LLMs can indeed produce diverse opinions according to the degree of task subjectivity.
arXiv Detail & Related papers (2023-11-16T11:23:38Z) - On the steerability of large language models toward data-driven personas [98.9138902560793]
Large language models (LLMs) are known to generate biased responses where the opinions of certain groups and populations are underrepresented.
Here, we present a novel approach to achieve controllable generation of specific viewpoints using LLMs.
arXiv Detail & Related papers (2023-11-08T19:01:13Z) - Editing Personality for Large Language Models [73.59001811199823]
This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs)
We construct PersonalityEdit, a new benchmark dataset to address this task.
arXiv Detail & Related papers (2023-10-03T16:02:36Z) - Revisiting the Reliability of Psychological Scales on Large Language Models [62.57981196992073]
This study aims to determine the reliability of applying personality assessments to Large Language Models.
Analysis of 2,500 settings per model, including GPT-3.5, GPT-4, Gemini-Pro, and LLaMA-3.1, reveals that various LLMs show consistency in responses to the Big Five Inventory.
arXiv Detail & Related papers (2023-05-31T15:03:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.