Assessing the nature of large language models: A caution against anthropocentrism

• URL: http://arxiv.org/abs/2309.07683v3
• Date: Thu, 27 Jun 2024 15:54:58 GMT
• Title: Assessing the nature of large language models: A caution against anthropocentrism
• Authors: Ann Speed,
• Abstract summary: We assessed several LLMs, primarily GPT 3.5, using standard, normed, and validated cognitive and personality measures. Our results indicate that LLMs are unlikely to have developed sentience, although its ability to respond to personality inventories is interesting. GPT3.5 did display large variability in both cognitive and personality measures over repeated observations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI models garnered a large amount of public attention and speculation with the release of OpenAIs chatbot, ChatGPT. At least two opinion camps exist: one excited about possibilities these models offer for fundamental changes to human tasks, and another highly concerned about power these models seem to have. To address these concerns, we assessed several LLMs, primarily GPT 3.5, using standard, normed, and validated cognitive and personality measures. For this seedling project, we developed a battery of tests that allowed us to estimate the boundaries of some of these models capabilities, how stable those capabilities are over a short period of time, and how they compare to humans. Our results indicate that LLMs are unlikely to have developed sentience, although its ability to respond to personality inventories is interesting. GPT3.5 did display large variability in both cognitive and personality measures over repeated observations, which is not expected if it had a human-like personality. Variability notwithstanding, LLMs display what in a human would be considered poor mental health, including low self-esteem, marked dissociation from reality, and in some cases narcissism and psychopathy, despite upbeat and helpful responses.

Related papers

• Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models [57.518784855080334]
  Large Language Models (LLMs) have demonstrated exceptional task-solving capabilities, increasingly adopting roles akin to human-like assistants. This paper presents a framework for investigating psychology dimension in LLMs, including psychological identification, assessment dataset curation, and assessment with results validation. We introduce a comprehensive psychometrics benchmark for LLMs that covers six psychological dimensions: personality, values, emotion, theory of mind, motivation, and intelligence.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• Limited Ability of LLMs to Simulate Human Psychological Behaviours: a Psychometric Analysis [0.27309692684728604]
  We prompt OpenAI's flagship models, GPT-3.5 and GPT-4, to assume different personas and respond to a range of standardized measures of personality constructs. We found that the responses from GPT-4, but not GPT-3.5, using generic persona descriptions show promising, albeit not perfect, psychometric properties.
  arXiv  Detail & Related papers  (2024-05-12T10:52:15Z)
• Large Language Models Show Human-like Social Desirability Biases in Survey Responses [12.767606361552684]
  We show that Large Language Models (LLMs) skew their scores towards the desirable ends of trait dimensions when personality evaluation is inferred. This bias exists in all tested models, including GPT-4/3.5, Claude 3, Llama 3, and PaLM-2. reverse-coding all the questions decreases bias levels but does not eliminate them, suggesting that this effect cannot be attributed to acquiescence bias.
  arXiv  Detail & Related papers  (2024-05-09T19:02:53Z)
• Theory of Mind abilities of Large Language Models in Human-Robot Interaction : An Illusion? [18.770522926093786]
  Large Language Models have shown exceptional generative abilities in various natural language and generation tasks. We study a special application of ToM abilities that has higher stakes and possibly irreversible consequences. We focus on the task of Perceived Behavior Recognition, where a robot employs a Large Language Model (LLM) to assess the robot's generated behavior in a manner similar to human observer.
  arXiv  Detail & Related papers  (2024-01-10T18:09:36Z)
• Do LLMs exhibit human-like response biases? A case study in survey design [66.1850490474361]
  We investigate the extent to which large language models (LLMs) reflect human response biases, if at all. We design a dataset and framework to evaluate whether LLMs exhibit human-like response biases in survey questionnaires. Our comprehensive evaluation of nine models shows that popular open and commercial LLMs generally fail to reflect human-like behavior.
  arXiv  Detail & Related papers  (2023-11-07T15:40:43Z)
• Editing Personality for Large Language Models [73.59001811199823]
  This paper introduces an innovative task focused on editing the personality traits of Large Language Models (LLMs) We construct a new benchmark dataset PersonalityEdit to address this task.
  arXiv  Detail & Related papers  (2023-10-03T16:02:36Z)
• Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBench [83.41621219298489]
  We propose to evaluate the empathy ability of Large Language Models (LLMs) We collect a dataset containing over 400 situations that have proven effective in eliciting the eight emotions central to our study. We conduct a human evaluation involving more than 1,200 subjects worldwide.
  arXiv  Detail & Related papers  (2023-08-07T15:18:30Z)
• Human-Like Intuitive Behavior and Reasoning Biases Emerged in Language Models -- and Disappeared in GPT-4 [0.0]
  We show that large language models (LLMs) exhibit behavior that resembles human-like intuition. We also probe how sturdy the inclination for intuitive-like decision-making is.
  arXiv  Detail & Related papers  (2023-06-13T08:43:13Z)
• Revisiting the Reliability of Psychological Scales on Large Language Models [66.31055885857062]
  This study aims to determine the reliability of applying personality assessments to Large Language Models (LLMs) By shedding light on the personalization of LLMs, our study endeavors to pave the way for future explorations in this field.
  arXiv  Detail & Related papers  (2023-05-31T15:03:28Z)
• Thinking Fast and Slow in Large Language Models [0.08057006406834465]
  Large language models (LLMs) are currently at the forefront of intertwining AI systems with human communication and everyday life. In this study, we show that LLMs like GPT-3 exhibit behavior that resembles human-like intuition - and the cognitive errors that come with it.
  arXiv  Detail & Related papers  (2022-12-10T05:07:30Z)
• Evaluating and Inducing Personality in Pre-trained Language Models [78.19379997967191]
  We draw inspiration from psychometric studies by leveraging human personality theory as a tool for studying machine behaviors. To answer these questions, we introduce the Machine Personality Inventory (MPI) tool for studying machine behaviors. MPI follows standardized personality tests, built upon the Big Five Personality Factors (Big Five) theory and personality assessment inventories. We devise a Personality Prompting (P2) method to induce LLMs with specific personalities in a controllable way.
  arXiv  Detail & Related papers  (2022-05-20T07:32:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.