R.U.Psycho? Robust Unified Psychometric Testing of Language Models

• URL: http://arxiv.org/abs/2503.10229v1
• Date: Thu, 13 Mar 2025 10:12:34 GMT
• Title: R.U.Psycho? Robust Unified Psychometric Testing of Language Models
• Authors: Julian Schelb, Orr Borin, David Garcia, Andreas Spitz,
• Abstract summary: R.U.Psycho is a framework for designing and running robust and reproducible psychometric experiments on generative language models.<n>We demonstrate the capability of our framework on a variety of psychometric questionnaires, which lend support to prior findings in the literature.
• Score: 3.3579972443598214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative language models are increasingly being subjected to psychometric questionnaires intended for human testing, in efforts to establish their traits, as benchmarks for alignment, or to simulate participants in social science experiments. While this growing body of work sheds light on the likeness of model responses to those of humans, concerns are warranted regarding the rigour and reproducibility with which these experiments may be conducted. Instabilities in model outputs, sensitivity to prompt design, parameter settings, and a large number of available model versions increase documentation requirements. Consequently, generalization of findings is often complex and reproducibility is far from guaranteed. In this paper, we present R.U.Psycho, a framework for designing and running robust and reproducible psychometric experiments on generative language models that requires limited coding expertise. We demonstrate the capability of our framework on a variety of psychometric questionnaires, which lend support to prior findings in the literature. R.U.Psycho is available as a Python package at https://github.com/julianschelb/rupsycho.

Related papers

• Reverse-Engineering the Reader [43.26660964074272]
  We introduce a novel alignment technique in which we fine-tune a language model to implicitly optimize the parameters of a linear regressor. Using words as a test case, we evaluate our technique across multiple model sizes and datasets. We find an inverse relationship between psychometric power and a model's performance on downstream NLP tasks as well as its perplexity on held-out test data.
  arXiv  Detail & Related papers  (2024-10-16T23:05:01Z)
• Assessment and manipulation of latent constructs in pre-trained language models using psychometric scales [4.805861461250903]
  We show how standard psychological questionnaires can be reformulated into natural language inference prompts.<n>We demonstrate, using a sample of 88 publicly available models, the existence of human-like mental health-related constructs.
  arXiv  Detail & Related papers  (2024-09-29T11:00:41Z)
• Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics [1.1172147007388977]
  We provide a state of the art language model with the same personality questionnaire in nine languages. Our results suggest both interlingual and intralingual instabilities, which indicate that current language models do not develop a consistent core personality. This can lead to unsafe behaviour of artificial intelligence systems that are based on these foundation models.
  arXiv  Detail & Related papers  (2024-08-14T08:53:00Z)
• Nonparametric independence tests in high-dimensional settings, with applications to the genetics of complex disease [55.2480439325792]
  We show how defining adequate premetric structures on the support spaces of the genetic data allows for novel approaches to such testing. For each problem, we provide mathematical results, simulations and the application to real data.
  arXiv  Detail & Related papers  (2024-07-29T01:00:53Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• Estimating the Personality of White-Box Language Models [0.589889361990138]
  Large-scale language models, which are trained on large corpora of text, are being used in a wide range of applications everywhere. Existing research shows that these models can and do capture human biases. Many of these biases, especially those that could potentially cause harm, are being well-investigated. However, studies that infer and change human personality traits inherited by these models have been scarce or non-existent.
  arXiv  Detail & Related papers  (2022-04-25T23:53:53Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• PyHealth: A Python Library for Health Predictive Models [53.848478115284195]
  PyHealth is an open-source Python toolbox for developing various predictive models on healthcare data. The data preprocessing module enables the transformation of complex healthcare datasets into machine learning friendly formats. The predictive modeling module provides more than 30 machine learning models, including established ensemble trees and deep neural network-based approaches.
  arXiv  Detail & Related papers  (2021-01-11T22:02:08Z)
• On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior [29.260666424382446]
  We test over two dozen models on how well their next-word expectations predict human reading time on naturalistic text corpora. We evaluate how features of these models determine their psychometric predictive power, or ability to predict human reading behavior. For any given perplexity, deep Transformer models and n-gram models show superior psychometric predictive power over LSTM or structurally supervised neural models.
  arXiv  Detail & Related papers  (2020-06-02T19:47:01Z)
• Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples [84.8370546614042]
  Black-box nature of Deep Learning models has posed unanswered questions about what they learn from data. Generative Adversarial Network (GAN) and multi-objectives are used to furnish a plausible attack to the audited model. Its utility is showcased within a human face classification task, unveiling the enormous potential of the proposed framework.
  arXiv  Detail & Related papers  (2020-03-25T11:08:56Z)
• Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study [81.11161697133095]
  We take the NER task as a testbed to analyze the generalization behavior of existing models from different perspectives. Experiments with in-depth analyses diagnose the bottleneck of existing neural NER models. As a by-product of this paper, we have open-sourced a project that involves a comprehensive summary of recent NER papers.
  arXiv  Detail & Related papers  (2020-01-12T04:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.