Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics

• URL: http://arxiv.org/abs/2408.07377v2
• Date: Thu, 15 Aug 2024 05:15:55 GMT
• Title: Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics
• Authors: Peter Romero, Stephen Fitz, Teruo Nakatsuma,
• Abstract summary: We provide a state of the art language model with the same personality questionnaire in nine languages. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models.
• Score: 1.1172147007388977
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Previous research on emergence in large language models shows these display apparent human-like abilities and psychological latent traits. However, results are partly contradicting in expression and magnitude of these latent traits, yet agree on the worrisome tendencies to score high on the Dark Triad of narcissism, psychopathy, and Machiavellianism, which, together with a track record of derailments, demands more rigorous research on safety of these models. We provided a state of the art language model with the same personality questionnaire in nine languages, and performed Bayesian analysis of Gaussian Mixture Model, finding evidence for a deeper-rooted issue. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models, and are increasingly integrated in human life. We subsequently discuss the shortcomings of modern psychometrics, abstract it, and provide a framework for its species-neutral, substrate-free formulation.

Related papers

• Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning [84.94709351266557]
  We focus on the trustworthiness of language models with respect to retrieval augmentation. We deem that retrieval-augmented language models have the inherent capabilities of supplying response according to both contextual and parametric knowledge. Inspired by aligning language models with human preference, we take the first step towards aligning retrieval-augmented language models to a status where it responds relying merely on the external evidence.
  arXiv  Detail & Related papers  (2024-10-22T09:25:21Z)
• On the Proper Treatment of Tokenization in Psycholinguistics [53.960910019072436]
  The paper argues that token-level language models should be marginalized into character-level language models before they are used in psycholinguistic studies. We find various focal areas whose surprisal is a better psychometric predictor than the surprisal of the region of interest itself.
  arXiv  Detail & Related papers  (2024-10-03T17:18:03Z)
• Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales [4.805861461250903]
  We show how standard psychological questionnaires can be reformulated into natural language inference prompts. We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs.
  arXiv  Detail & Related papers  (2024-09-29T11:00:41Z)
• The Devil is in the Neurons: Interpreting and Mitigating Social Biases in Pre-trained Language Models [78.69526166193236]
  Pre-trained Language models (PLMs) have been acknowledged to contain harmful information, such as social biases. We propose sc Social Bias Neurons to accurately pinpoint units (i.e., neurons) in a language model that can be attributed to undesirable behavior, such as social bias. As measured by prior metrics from StereoSet, our model achieves a higher degree of fairness while maintaining language modeling ability with low cost.
  arXiv  Detail & Related papers  (2024-06-14T15:41:06Z)
• Editing Personality for Large Language Models [73.59001811199823]
  This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs) We construct PersonalityEdit, a new benchmark dataset to address this task.
  arXiv  Detail & Related papers  (2023-10-03T16:02:36Z)
• Out of One, Many: Using Language Models to Simulate Human Samples [3.278541277919869]
  We show that the "algorithmic bias" within one such tool -- the GPT-3 language model -- is both fine-grained and demographically correlated. We create "silicon samples" by conditioning the model on thousands of socio-demographic backstories from real human participants.
  arXiv  Detail & Related papers  (2022-09-14T19:53:32Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Estimating the Personality of White-Box Language Models [0.589889361990138]
  Large-scale language models, which are trained on large corpora of text, are being used in a wide range of applications everywhere. Existing research shows that these models can and do capture human biases. Many of these biases, especially those that could potentially cause harm, are being well-investigated. However, studies that infer and change human personality traits inherited by these models have been scarce or non-existent.
  arXiv  Detail & Related papers  (2022-04-25T23:53:53Z)
• Schr\"odinger's Tree -- On Syntax and Neural Language Models [10.296219074343785]
  Language models have emerged as NLP's workhorse, displaying increasingly fluent generation capabilities. We observe a lack of clarity across numerous dimensions, which influences the hypotheses that researchers form. We outline the implications of the different types of research questions exhibited in studies on syntax.
  arXiv  Detail & Related papers  (2021-10-17T18:25:23Z)
• Uncovering Constraint-Based Behavior in Neural Models via Targeted Fine-Tuning [9.391375268580806]
  We show that competing linguistic processes within a language obscure underlying linguistic knowledge. While human behavior has been found to be similar across languages, we find cross-linguistic variation in model behavior. Our results suggest that models need to learn both the linguistic constraints in a language and their relative ranking, with mismatches in either producing non-human-like behavior.
  arXiv  Detail & Related papers  (2021-06-02T14:52:11Z)
• Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
  We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
  arXiv  Detail & Related papers  (2020-06-19T12:00:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.