AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities

• URL: http://arxiv.org/abs/2603.11279v1
• Date: Wed, 11 Mar 2026 20:15:56 GMT
• Title: AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities
• Abstract summary: The immense number of parameters and deep neural networks make large language models (LLMs) rival the complexity of human brains.<n>This paper investigates the application of AI Psychometrics to evaluate the psychological reasoning and overall psychometric validity of four prominent LLMs.
• Score: 5.532324061241031
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The immense number of parameters and deep neural networks make large language models (LLMs) rival the complexity of human brains, which also makes them opaque ``black box'' systems that are challenging to evaluate and interpret. AI Psychometrics is an emerging field that aims to tackle these challenges by applying psychometric methodologies to evaluate and interpret the psychological traits and processes of artificial intelligence (AI) systems. This paper investigates the application of AI Psychometrics to evaluate the psychological reasoning and overall psychometric validity of four prominent LLMs: GPT-3.5, GPT-4, LLaMA-2, and LLaMA-3. Using the Technology Acceptance Model (TAM), we examined convergent, discriminant, predictive, and external validity across these models. Our findings reveal that the responses from all these models generally met all validity criteria. Moreover, higher-performing models like GPT-4 and LLaMA-3 consistently demonstrated superior psychometric validity compared to their predecessors, GPT-3.5 and LLaMA-2. These results help to establish the validity of applying AI Psychometrics to evaluate and interpret large language models.

Related papers

• A Survey of Large Language Models for Perception and Measurement of Human Psychology [55.608047817598965]
  A central issue is whether Large Language Models possess the capability to accurately perceive and measure complex, latent human psychological constructs.<n>This paper provides a systematic review focused on the use of LLMs as instruments for human psychological measurement.
  arXiv  Detail & Related papers  (2026-05-20T14:54:09Z)
• Projective Psychological Assessment of Large Multimodal Models Using Thematic Apperception Tests [5.119837168333715]
  This study examines whether the personality traits of Large Multimodal Models (LMMs) can be assessed through non-language-based modalities.<n>Evaluators demonstrated an excellent ability to understand and analyze TAT responses.
  arXiv  Detail & Related papers  (2026-02-19T06:08:33Z)
• HUMANLLM: Benchmarking and Reinforcing LLM Anthropomorphism via Human Cognitive Patterns [59.17423586203706]
  We present HUMANLLM, a framework treating psychological patterns as interacting causal forces.<n>We construct 244 patterns from 12,000 academic papers and synthesize 11,359 scenarios where 2-5 patterns reinforce, conflict, or modulate each other.<n>Our dual-level checklists evaluate both individual pattern fidelity and emergent multi-pattern dynamics, achieving strong human alignment.
  arXiv  Detail & Related papers  (2026-01-15T08:56:53Z)
• MindEval: Benchmarking Language Models on Multi-turn Mental Health Support [10.524387723320432]
  MindEval is a framework for automatically evaluating language models in realistic, multi-turn mental health therapy conversations.<n>We quantitatively validate the realism of our simulated patients against human-generated text and by demonstrating strong correlations between automatic and human expert judgments.<n>We evaluate 12 state-of-the-art LLMs and show that all models struggle, scoring below 4 out of 6 on average, with particular weaknesses in problematic AI-specific patterns of communication.
  arXiv  Detail & Related papers  (2025-11-23T15:19:29Z)
• Measuring How LLMs Internalize Human Psychological Concepts: A preliminary analysis [0.0]
  We develop a framework to assess concept alignment between Large Language Models and human psychological dimensions.<n>A GPT-4 model achieved superior classification accuracy (66.2%), significantly outperforming GPT-3.5 (55.9%) and BERT (48.1%)<n>Our findings demonstrate that modern LLMs can approximate human psychological constructs with measurable accuracy.
  arXiv  Detail & Related papers  (2025-06-29T01:56:56Z)
• Beyond Empathy: Integrating Diagnostic and Therapeutic Reasoning with Large Language Models for Mental Health Counseling [50.83055329849865]
  PsyLLM is a large language model designed to integrate diagnostic and therapeutic reasoning for mental health counseling.<n>It processes real-world mental health posts from Reddit and generates multi-turn dialogue structures.<n>Our experiments demonstrate that PsyLLM significantly outperforms state-of-the-art baseline models.
  arXiv  Detail & Related papers  (2025-05-21T16:24:49Z)
• Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement [16.608577295968942]
  Review paper introduces and synthesizes the emerging interdisciplinary field of LLM Psychometrics.<n>Psychometrics quantify intangible aspects of human psychology, such as personality, values, and intelligence.<n>Ultimately, the review provides actionable insights for developing future evaluation paradigms that align with human-level AI.
  arXiv  Detail & Related papers  (2025-05-13T05:47:51Z)
• Evaluating Large Language Models with Psychometrics [59.821829073478376]
  This paper offers a comprehensive benchmark for quantifying psychological constructs of Large Language Models (LLMs)<n>Our work identifies five key psychological constructs -- personality, values, emotional intelligence, theory of mind, and self-efficacy -- assessed through a suite of 13 datasets.<n>We uncover significant discrepancies between LLMs' self-reported traits and their response patterns in real-world scenarios, revealing complexities in their behaviors.
  arXiv  Detail & Related papers  (2024-06-25T16:09:08Z)
• Machine Psychology [54.287802134327485]
  We argue that a fruitful direction for research is engaging large language models in behavioral experiments inspired by psychology. We highlight theoretical perspectives, experimental paradigms, and computational analysis techniques that this approach brings to the table. It paves the way for a "machine psychology" for generative artificial intelligence (AI) that goes beyond performance benchmarks.
  arXiv  Detail & Related papers  (2023-03-24T13:24:41Z)
• Evaluating Psychological Safety of Large Language Models [72.88260608425949]
  We designed unbiased prompts to evaluate the psychological safety of large language models (LLMs) We tested five different LLMs by using two personality tests: Short Dark Triad (SD-3) and Big Five Inventory (BFI) Despite being instruction fine-tuned with safety metrics to reduce toxicity, InstructGPT, GPT-3.5, and GPT-4 still showed dark personality patterns. Fine-tuning Llama-2-chat-7B with responses from BFI using direct preference optimization could effectively reduce the psychological toxicity of the model.
  arXiv  Detail & Related papers  (2022-12-20T18:45:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.