Is ChatGPT a Good Personality Recognizer? A Preliminary Study
- URL: http://arxiv.org/abs/2307.03952v3
- Date: Thu, 28 Dec 2023 16:43:59 GMT
- Title: Is ChatGPT a Good Personality Recognizer? A Preliminary Study
- Authors: Yu Ji, Wen Wu, Hong Zheng, Yi Hu, Xi Chen, Liang He
- Abstract summary: This study investigates ChatGPT's ability in recognizing personality from given text.
We employ a variety of prompting strategies to explore ChatGPT's ability in recognizing personality from given text.
- Score: 19.278538849802025
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In recent years, personality has been regarded as a valuable personal factor
being incorporated into numerous tasks such as sentiment analysis and product
recommendation. This has led to widespread attention to text-based personality
recognition task, which aims to identify an individual's personality based on
given text. Considering that ChatGPT has recently exhibited remarkable
abilities on various natural language processing tasks, we provide a
preliminary evaluation of ChatGPT on text-based personality recognition task
for generating effective personality data. Concretely, we employ a variety of
prompting strategies to explore ChatGPT's ability in recognizing personality
from given text, especially the level-oriented prompting strategy we designed
for guiding ChatGPT in analyzing given text at a specified level. The
experimental results on two representative real-world datasets reveal that
ChatGPT with zero-shot chain-of-thought prompting exhibits impressive
personality recognition ability and is capable to provide natural language
explanations through text-based logical reasoning. Furthermore, by employing
the level-oriented prompting strategy to optimize zero-shot chain-of-thought
prompting, the performance gap between ChatGPT and corresponding
state-of-the-art model has been narrowed even more. However, we observe that
ChatGPT shows unfairness towards certain sensitive demographic attributes such
as gender and age. Additionally, we discover that eliciting the personality
recognition ability of ChatGPT helps improve its performance on
personality-related downstream tasks such as sentiment classification and
stress prediction.
Related papers
- Can ChatGPT Read Who You Are? [10.577227353680994]
We report the results of a comprehensive user study featuring texts written in Czech by a representative population sample of 155 participants.
We compare the personality trait estimations made by ChatGPT against those by human raters and report ChatGPT's competitive performance in inferring personality traits from text.
arXiv Detail & Related papers (2023-12-26T14:43:04Z) - PsyCoT: Psychological Questionnaire as Powerful Chain-of-Thought for
Personality Detection [50.66968526809069]
We propose a novel personality detection method, called PsyCoT, which mimics the way individuals complete psychological questionnaires in a multi-turn dialogue manner.
Our experiments demonstrate that PsyCoT significantly improves the performance and robustness of GPT-3.5 in personality detection.
arXiv Detail & Related papers (2023-10-31T08:23:33Z) - Is ChatGPT Involved in Texts? Measure the Polish Ratio to Detect
ChatGPT-Generated Text [48.36706154871577]
We introduce a novel dataset termed HPPT (ChatGPT-polished academic abstracts)
It diverges from extant corpora by comprising pairs of human-written and ChatGPT-polished abstracts instead of purely ChatGPT-generated texts.
We also propose the "Polish Ratio" method, an innovative measure of the degree of modification made by ChatGPT compared to the original human-written text.
arXiv Detail & Related papers (2023-07-21T06:38:37Z) - ChatGPT vs Human-authored Text: Insights into Controllable Text
Summarization and Sentence Style Transfer [8.64514166615844]
We conduct a systematic inspection of ChatGPT's performance in two controllable generation tasks.
We evaluate the faithfulness of the generated text, and compare the model's performance with human-authored texts.
We observe that ChatGPT sometimes incorporates factual errors or hallucinations when adapting the text to suit a specific style.
arXiv Detail & Related papers (2023-06-13T14:21:35Z) - To ChatGPT, or not to ChatGPT: That is the question! [78.407861566006]
This study provides a comprehensive and contemporary assessment of the most recent techniques in ChatGPT detection.
We have curated a benchmark dataset consisting of prompts from ChatGPT and humans, including diverse questions from medical, open Q&A, and finance domains.
Our evaluation results demonstrate that none of the existing methods can effectively detect ChatGPT-generated content.
arXiv Detail & Related papers (2023-04-04T03:04:28Z) - Is ChatGPT A Good Keyphrase Generator? A Preliminary Study [51.863368917344864]
ChatGPT has recently garnered significant attention from the computational linguistics community.
We evaluate its performance in various aspects, including keyphrase generation prompts, keyphrase generation diversity, and long document understanding.
We find that ChatGPT performs exceptionally well on all six candidate prompts, with minor performance differences observed across the datasets.
arXiv Detail & Related papers (2023-03-23T02:50:38Z) - Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on
Consistency with Human Preferences [6.821378903525802]
ChatGPT has consistently demonstrated a remarkable level of accuracy and reliability in terms of content evaluation.
A test set consisting of prompts is created, covering a wide range of use cases, and five models are utilized to generate corresponding responses.
Results on the test set show that ChatGPT's ranking preferences are consistent with human to a certain extent.
arXiv Detail & Related papers (2023-03-14T03:13:02Z) - ChatGPT: Jack of all trades, master of none [4.693597927153063]
OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT)
We examined ChatGPT's capabilities on 25 diverse analytical NLP tasks.
We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses.
arXiv Detail & Related papers (2023-02-21T15:20:37Z) - Can ChatGPT Understand Too? A Comparative Study on ChatGPT and
Fine-tuned BERT [103.57103957631067]
ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries.
We evaluate ChatGPT's understanding ability by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models.
We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question answering tasks.
arXiv Detail & Related papers (2023-02-19T12:29:33Z) - Is ChatGPT a General-Purpose Natural Language Processing Task Solver? [113.22611481694825]
Large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot.
Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community.
It is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot.
arXiv Detail & Related papers (2023-02-08T09:44:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.