Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales

• URL: http://arxiv.org/abs/2409.19655v1
• Date: Sun, 29 Sep 2024 11:00:41 GMT
• Title: Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales
• Authors: Maor Reuben, Ortal Slobodin, Aviad Elyshar, Idan-Chaim Cohen, Orna Braun-Lewensohn, Odeya Cohen, Rami Puzis,
• Abstract summary: We show how standard psychological questionnaires can be reformulated into natural language inference prompts. We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs.
• Score: 4.805861461250903
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Human-like personality traits have recently been discovered in large language models, raising the hypothesis that their (known and as yet undiscovered) biases conform with human latent psychological constructs. While large conversational models may be tricked into answering psychometric questionnaires, the latent psychological constructs of thousands of simpler transformers, trained for other tasks, cannot be assessed because appropriate psychometric methods are currently lacking. Here, we show how standard psychological questionnaires can be reformulated into natural language inference prompts, and we provide a code library to support the psychometric assessment of arbitrary models. We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs (including anxiety, depression, and Sense of Coherence) which conform with standard theories in human psychology and show similar correlations and mitigation strategies. The ability to interpret and rectify the performance of language models by using psychological tools can boost the development of more explainable, controllable, and trustworthy models.

Related papers

• MentalArena: Self-play Training of Language Models for Diagnosis and Treatment of Mental Health Disorders [59.515827458631975]
  Mental health disorders are one of the most serious diseases in the world. Privacy concerns limit the accessibility of personalized treatment data. MentalArena is a self-play framework to train language models.
  arXiv  Detail & Related papers  (2024-10-09T13:06:40Z)
• Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics [1.1172147007388977]
  We provide a state of the art language model with the same personality questionnaire in nine languages. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models.
  arXiv  Detail & Related papers  (2024-08-14T08:53:00Z)
• PersLLM: A Personified Training Approach for Large Language Models [66.16513246245401]
  We propose PersLLM, integrating psychology-grounded principles of personality: social practice, consistency, and dynamic development. We incorporate personality traits directly into the model parameters, enhancing the model's resistance to induction, promoting consistency, and supporting the dynamic evolution of personality.
  arXiv  Detail & Related papers  (2024-07-17T08:13:22Z)
• LLM Questionnaire Completion for Automatic Psychiatric Assessment [49.1574468325115]
  We employ a Large Language Model (LLM) to convert unstructured psychological interviews into structured questionnaires spanning various psychiatric and personality domains. The obtained answers are coded as features, which are used to predict standardized psychiatric measures of depression (PHQ-8) and PTSD (PCL-C)
  arXiv  Detail & Related papers  (2024-06-09T09:03:11Z)
• PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents [68.50571379012621]
  Psychological measurement is essential for mental health, self-understanding, and personal development. PsychoGAT (Psychological Game AgenTs) achieves statistically significant excellence in psychometric metrics such as reliability, convergent validity, and discriminant validity.
  arXiv  Detail & Related papers  (2024-02-19T18:00:30Z)
• Decoding Susceptibility: Modeling Misbelief to Misinformation Through a Computational Approach [61.04606493712002]
  Susceptibility to misinformation describes the degree of belief in unverifiable claims that is not observable. Existing susceptibility studies heavily rely on self-reported beliefs. We propose a computational approach to model users' latent susceptibility levels.
  arXiv  Detail & Related papers  (2023-11-16T07:22:56Z)
• Using Artificial Populations to Study Psychological Phenomena in Neural Models [0.0]
  Investigation of cognitive behavior in language models must be conducted in an appropriate population for the results to be meaningful. We leverage work in uncertainty estimation in a novel approach to efficiently construct experimental populations. We provide theoretical grounding in the uncertainty estimation literature and motivation from current cognitive work regarding language models.
  arXiv  Detail & Related papers  (2023-08-15T20:47:51Z)
• Predicting Human Psychometric Properties Using Computational Language Models [5.806723407090421]
  Transformer-based language models (LMs) continue to achieve state-of-the-art performance on natural language processing (NLP) benchmarks. Can LMs be of use in predicting the psychometric properties of test items, when those items are given to human participants? We gather responses from numerous human participants and LMs on a broad diagnostic test of linguistic competencies. We then use the human responses to calculate standard psychometric properties of the items in the diagnostic test, using the human responses and the LM responses separately.
  arXiv  Detail & Related papers  (2022-05-12T16:40:12Z)
• Towards Automated Psychotherapy via Language Modeling [0.0]
  The model was trained upon a mix of the Cornell Movie Dialogue Corpus for language understanding and an open-source, anonymized, and public licensed psychotherapeutic dataset. The model achieved statistically significant performance in published, standardized qualitative benchmarks against human-written validation data. Although the model cannot replace the work of psychotherapists entirely, its ability to synthesize human-appearing utterances for the majority of the test set serves as a promising step towards communizing and easing stigma at the psychotherapeutic point-of-care.
  arXiv  Detail & Related papers  (2021-04-05T01:53:39Z)
• AGENT: A Benchmark for Core Psychological Reasoning [60.35621718321559]
  Intuitive psychology is the ability to reason about hidden mental variables that drive observable actions. Despite recent interest in machine agents that reason about other agents, it is not clear if such agents learn or hold the core psychology principles that drive human reasoning. We present a benchmark consisting of procedurally generated 3D animations, AGENT, structured around four scenarios.
  arXiv  Detail & Related papers  (2021-02-24T14:58:23Z)
• On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior [29.260666424382446]
  We test over two dozen models on how well their next-word expectations predict human reading time on naturalistic text corpora. We evaluate how features of these models determine their psychometric predictive power, or ability to predict human reading behavior. For any given perplexity, deep Transformer models and n-gram models show superior psychometric predictive power over LSTM or structurally supervised neural models.
  arXiv  Detail & Related papers  (2020-06-02T19:47:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.