Mind Reading or Misreading? LLMs on the Big Five Personality Test

• URL: http://arxiv.org/abs/2511.23101v1
• Date: Fri, 28 Nov 2025 11:40:30 GMT
• Title: Mind Reading or Misreading? LLMs on the Big Five Personality Test
• Authors: Francesco Di Cursi, Chiara Boldrini, Marco Conti, Andrea Passarella,
• Abstract summary: We evaluate large language models (LLMs) for automatic personality prediction from text under the binary Five Factor Model (BIG5).<n>Open-source models sometimes approach GPT-4 and prior benchmarks, but no configuration yields consistently reliable predictions in zero-shot binary settings.<n>These findings show that current out-of-the-box LLMs are not yet suitable for APPT, and that careful coordination of prompt design, trait framing, and evaluation metrics is essential for interpretable results.
• Score: 1.3649494534428745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We evaluate large language models (LLMs) for automatic personality prediction from text under the binary Five Factor Model (BIG5). Five models -- including GPT-4 and lightweight open-source alternatives -- are tested across three heterogeneous datasets (Essays, MyPersonality, Pandora) and two prompting strategies (minimal vs. enriched with linguistic and psychological cues). Enriched prompts reduce invalid outputs and improve class balance, but also introduce a systematic bias toward predicting trait presence. Performance varies substantially: Openness and Agreeableness are relatively easier to detect, while Extraversion and Neuroticism remain challenging. Although open-source models sometimes approach GPT-4 and prior benchmarks, no configuration yields consistently reliable predictions in zero-shot binary settings. Moreover, aggregate metrics such as accuracy and macro-F1 mask significant asymmetries, with per-class recall offering clearer diagnostic value. These findings show that current out-of-the-box LLMs are not yet suitable for APPT, and that careful coordination of prompt design, trait framing, and evaluation metrics is essential for interpretable results.

Related papers

• Uncertainty and Fairness Awareness in LLM-Based Recommendation Systems [3.937681476010311]
  This paper studies how uncertainty and fairness evaluations affect the accuracy, consistency, and trustworthiness of large language models (LLMs)<n>We quantify predictive uncertainty (via entropy) and demonstrate that Google DeepMind's Gemini 1.5 Flash exhibits systematic unfairness for certain sensitive attributes.<n>We propose a novel uncertainty-aware evaluation methodology for RecLLMs, present empirical insights from deep uncertainty case studies, and introduce a personality profile-informed fairness benchmark.
  arXiv  Detail & Related papers  (2026-01-31T17:18:13Z)
• Evaluating Prompting Strategies and Large Language Models in Systematic Literature Review Screening: Relevance and Task-Stage Classification [1.2234742322758418]
  This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews.<n>We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, Llama-4-Maverick) under five prompt types.<n>CoT-few-shot yields the most reliable precision-recall balance; zero-shot maximizes recall for high-sensitivity passes; and self-reflection underperforms due to over-inclusivity and instability across models.
  arXiv  Detail & Related papers  (2025-10-17T16:53:09Z)
• DiSC-AMC: Token- and Parameter-Efficient Discretized Statistics In-Context Automatic Modulation Classification [19.190236060870184]
  Large Language Models (LLMs) can perform Automatic Modulation Classification (AMC) in an open-set manner without fine-tuning.<n>We present Discretized Statistics in-Context Automatic Modulation Classification (DiSC-AMC)<n>DiSC-AMC is a token- and parameter-efficient variant that discretizes higher-order statistics and cumulants into compact symbolic tokens.
  arXiv  Detail & Related papers  (2025-09-30T22:20:57Z)
• Evaluating LLM Alignment on Personality Inference from Real-World Interview Data [7.061237517845673]
  Large Language Models (LLMs) are increasingly deployed in roles requiring nuanced psychological understanding.<n>Their ability to interpret human personality traits, a critical aspect of such applications, remains unexplored.<n>We introduce a novel benchmark comprising semi-structured interview transcripts paired with validated continuous Big Five trait scores.
  arXiv  Detail & Related papers  (2025-09-16T16:54:35Z)
• When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs [55.20230501807337]
  We present the first systematic evaluation of 5 methods for improving prompt robustness within a unified experimental framework.<n>We benchmark these techniques on 8 models from Llama, Qwen and Gemma families across 52 tasks from Natural Instructions dataset.
  arXiv  Detail & Related papers  (2025-08-15T10:32:50Z)
• Can LLMs Infer Personality from Real World Conversations? [5.705775078773656]
  Large Language Models (LLMs) offer a promising approach for scalable personality assessment from open-ended language.<n>Three state-of-the-art LLMs were tested using zero-shot prompting for BFI-10 item prediction and both zero-shot and chain-of-thought prompting for Big Five trait inference.<n>All models showed high test-retest reliability, but construct validity was limited.
  arXiv  Detail & Related papers  (2025-07-18T20:22:47Z)
• Debiased Prompt Tuning in Vision-Language Model without Annotations [14.811475313694041]
  Vision-Language Models (VLMs) may suffer from the problem of spurious correlations.<n>By leveraging pseudo-spurious attribute annotations, we propose a method to automatically adjust the training weights of different groups.<n>Our approach efficiently improves the worst-group accuracy on CelebA, Waterbirds, and MetaShift datasets.
  arXiv  Detail & Related papers  (2025-03-11T12:24:54Z)
• How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts [54.07541591018305]
  We present MAD-Bench, a benchmark that contains 1000 test samples divided into 5 categories, such as non-existent objects, count of objects, and spatial relationship. We provide a comprehensive analysis of popular MLLMs, ranging from GPT-4v, Reka, Gemini-Pro, to open-sourced models, such as LLaVA-NeXT and MiniCPM-Llama3. While GPT-4o achieves 82.82% accuracy on MAD-Bench, the accuracy of any other model in our experiments ranges from 9% to 50%.
  arXiv  Detail & Related papers  (2024-02-20T18:31:27Z)
• M-Tuning: Prompt Tuning with Mitigated Label Bias in Open-Set Scenarios [58.617025733655005]
  We propose a vision-language prompt tuning method with mitigated label bias (M-Tuning)<n>It introduces open words from the WordNet to extend the range of words forming the prompt texts from only closed-set label words to more, and thus prompts are tuned in a simulated open-set scenario.<n>Our method achieves the best performance on datasets with various scales, and extensive ablation studies also validate its effectiveness.
  arXiv  Detail & Related papers  (2023-03-09T09:05:47Z)
• Prompting GPT-3 To Be Reliable [117.23966502293796]
  This work decomposes reliability into four facets: generalizability, fairness, calibration, and factuality. We find that GPT-3 outperforms smaller-scale supervised models by large margins on all these facets.
  arXiv  Detail & Related papers  (2022-10-17T14:52:39Z)
• Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases [55.45617404586874]
  We propose a few-shot instruction-based method for prompting pre-trained language models (LMs) We show that large LMs can detect different types of fine-grained biases with similar and sometimes superior accuracy to fine-tuned models.
  arXiv  Detail & Related papers  (2021-12-15T04:19:52Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.