Do LLMs Possess a Personality? Making the MBTI Test an Amazing
Evaluation for Large Language Models
- URL: http://arxiv.org/abs/2307.16180v1
- Date: Sun, 30 Jul 2023 09:34:35 GMT
- Title: Do LLMs Possess a Personality? Making the MBTI Test an Amazing
Evaluation for Large Language Models
- Authors: Keyu Pan, Yawen Zeng
- Abstract summary: We aim to investigate the feasibility of using the Myers-Briggs Type Indicator (MBTI), a widespread human personality assessment tool, as an evaluation metric for large language models (LLMs)
Specifically, experiments will be conducted to explore: 1) the personality types of different LLMs, 2) the possibility of changing the personality types by prompt engineering, and 3) How does the training dataset affect the model's personality.
- Score: 2.918940961856197
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The field of large language models (LLMs) has made significant progress, and
their knowledge storage capacity is approaching that of human beings.
Furthermore, advanced techniques, such as prompt learning and reinforcement
learning, are being employed to address ethical concerns and hallucination
problems associated with LLMs, bringing them closer to aligning with human
values. This situation naturally raises the question of whether LLMs with
human-like abilities possess a human-like personality? In this paper, we aim to
investigate the feasibility of using the Myers-Briggs Type Indicator (MBTI), a
widespread human personality assessment tool, as an evaluation metric for LLMs.
Specifically, extensive experiments will be conducted to explore: 1) the
personality types of different LLMs, 2) the possibility of changing the
personality types by prompt engineering, and 3) How does the training dataset
affect the model's personality. Although the MBTI is not a rigorous assessment,
it can still reflect the similarity between LLMs and human personality. In
practice, the MBTI has the potential to serve as a rough indicator. Our codes
are available at
https://github.com/HarderThenHarder/transformers_tasks/tree/main/LLM/llms_mbti.
Related papers
- The Better Angels of Machine Personality: How Personality Relates to LLM Safety [46.30207266304056]
The relationship between personality traits and safety abilities in Large Language Models (LLMs) remains a mystery.
We discover that LLMs' personality traits are closely related to their safety abilities.
inducing personality from ISTJ to ISTP resulted in a relative improvement of approximately 43% and 10% in privacy and fairness performance.
arXiv Detail & Related papers (2024-07-17T06:36:29Z) - Quantifying AI Psychology: A Psychometrics Benchmark for Large Language Models [57.518784855080334]
Large Language Models (LLMs) have demonstrated exceptional task-solving capabilities, increasingly adopting roles akin to human-like assistants.
This paper presents a framework for investigating psychology dimension in LLMs, including psychological identification, assessment dataset curation, and assessment with results validation.
We introduce a comprehensive psychometrics benchmark for LLMs that covers six psychological dimensions: personality, values, emotion, theory of mind, motivation, and intelligence.
arXiv Detail & Related papers (2024-06-25T16:09:08Z) - Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics [29.325576963215163]
The idea of personality in psychology, traditionally defined through observable behavior, has now been extended to Large Language Models (LLMs) to better understand their behavior.
Existing self-assessment personality tests, while applicable, lack the necessary validity and reliability for precise personality measurements.
We introduce TRAIT, a new tool consisting of 8K multi-choice questions designed to assess the personality of LLMs with validity and reliability.
arXiv Detail & Related papers (2024-06-20T19:50:56Z) - PHAnToM: Personality Has An Effect on Theory-of-Mind Reasoning in Large Language Models [25.657579792829743]
This study investigates how inducing personalities in large language models with prompts affects their Theory-of-Mind (ToM) reasoning capabilities.
We find that certain induced personalities can significantly affect the LLMs' reasoning capabilities in three different ToM tasks.
We find that LLMs that exhibit a higher variance across personality prompts in ToM also tends to be more controllable in personality tests.
arXiv Detail & Related papers (2024-03-04T17:34:34Z) - Identifying Multiple Personalities in Large Language Models with
External Evaluation [6.657168333238573]
Large Language Models (LLMs) are integrated with human daily applications rapidly.
Many recent studies quantify LLMs' personalities using self-assessment tests that are created for humans.
Yet many critiques question the applicability and reliability of these self-assessment tests when applied to LLMs.
arXiv Detail & Related papers (2024-02-22T18:57:20Z) - Illuminating the Black Box: A Psychometric Investigation into the
Multifaceted Nature of Large Language Models [3.692410936160711]
This study explores the idea of AI Personality or AInality suggesting that Large Language Models (LLMs) exhibit patterns similar to human personalities.
Using projective tests, we uncover hidden aspects of LLM personalities that are not easily accessible through direct questioning.
Our machine learning analysis revealed that LLMs exhibit distinct AInality traits and manifest diverse personality types, demonstrating dynamic shifts in response to external instructions.
arXiv Detail & Related papers (2023-12-21T04:57:21Z) - Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using
PsychoBench [83.41621219298489]
We propose a framework, PsychoBench, for evaluating diverse psychological aspects of Large Language Models (LLMs)
PsychoBench classifies these scales into four distinct categories: personality traits, interpersonal relationships, motivational tests, and emotional abilities.
We employ a jailbreak approach to bypass the safety alignment protocols and test the intrinsic natures of LLMs.
arXiv Detail & Related papers (2023-10-02T17:46:09Z) - Revisiting the Reliability of Psychological Scales on Large Language
Models [66.31055885857062]
This study aims to determine the reliability of applying personality assessments to Large Language Models (LLMs)
By shedding light on the personalization of LLMs, our study endeavors to pave the way for future explorations in this field.
arXiv Detail & Related papers (2023-05-31T15:03:28Z) - Can ChatGPT Assess Human Personalities? A General Evaluation Framework [70.90142717649785]
Large Language Models (LLMs) have produced impressive results in various areas, but their potential human-like psychology is still largely unexplored.
This paper presents a generic evaluation framework for LLMs to assess human personalities based on Myers Briggs Type Indicator (MBTI) tests.
arXiv Detail & Related papers (2023-03-01T06:16:14Z) - Evaluating and Inducing Personality in Pre-trained Language Models [78.19379997967191]
We draw inspiration from psychometric studies by leveraging human personality theory as a tool for studying machine behaviors.
To answer these questions, we introduce the Machine Personality Inventory (MPI) tool for studying machine behaviors.
MPI follows standardized personality tests, built upon the Big Five Personality Factors (Big Five) theory and personality assessment inventories.
We devise a Personality Prompting (P2) method to induce LLMs with specific personalities in a controllable way.
arXiv Detail & Related papers (2022-05-20T07:32:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.