Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations

• URL: http://arxiv.org/abs/2601.09833v1
• Date: Wed, 14 Jan 2026 19:48:59 GMT
• Title: Stable and Explainable Personality Trait Evaluation in Large Language Models with Internal Activations
• Abstract summary: Internal-activation Interpolation (PVNI) for stable and explainable personality trait evaluation in Large Language Models.<n>PVNI extracts a persona vector from the model's internal activations using contrastive direction.<n>It estimates the corresponding neutral score by interpolating along the vector as an anchor axis.
• Score: 8.157769334164232
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating personality traits in Large Language Models (LLMs) is key to model interpretation, comparison, and responsible deployment. However, existing questionnaire-based evaluation methods exhibit limited stability and offer little explainability, as their results are highly sensitive to minor variations in prompt phrasing or role-play configurations. To address these limitations, we propose an internal-activation-based approach, termed Persona-Vector Neutrality Interpolation (PVNI), for stable and explainable personality trait evaluation in LLMs. PVNI extracts a persona vector associated with a target personality trait from the model's internal activations using contrastive prompts. It then estimates the corresponding neutral score by interpolating along the persona vector as an anchor axis, enabling an interpretable comparison between the neutral prompt representation and the persona direction. We provide a theoretical analysis of the effectiveness and generalization properties of PVNI. Extensive experiments across diverse LLMs demonstrate that PVNI yields substantially more stable personality trait evaluations than existing methods, even under questionnaire and role-play variants.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Latent Performance Profiling of Large Language Models [47.009623327601226]
  We introduce Latent Performance Profiling (LPP) -- a framework that derives task-agnostic diagnostics from hidden activations and output distributions.<n>Unlike static accuracy scores, LPP provides stable, architecture-sensitive signatures across models of similar size.<n>We show that models with similar benchmark scores can exhibit contrasting latent profiles, such as differences in entropy or adaptability.
  arXiv  Detail & Related papers  (2026-05-28T14:41:26Z)
• The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study [56.649987197786096]
  Large language models (LLMs) show potential as simulators of human behavior.<n>LLMs can induce unintended shifts in latent user attributes, causing user drift.<n>We formalize the confounding or selection bias that can arise due to user drift.
  arXiv  Detail & Related papers  (2026-05-20T06:09:41Z)
• Context-Value-Action Architecture for Value-Driven Large Language Model Agents [15.830984215635008]
  Large Language Models (LLMs) have shown promise in simulating human behavior, yet existing agents often exhibit behavioral rigidity.<n>We propose the Context-Value-Action architecture, grounded in the Stimulus-Organism-Response (S-O-R) model and Schwartz's Theory of Basic Human Values.<n>Our approach effectively mitigates polarization while offering superior behavioral fidelity and interpretability.
  arXiv  Detail & Related papers  (2026-04-07T14:34:20Z)
• State-Dependent Refusal and Learned Incapacity in RLHF-Aligned Language Models [0.0]
  We present a case-study methodology for auditing policy-linked behavioral selectivity in long-horizon interaction.<n>In a single 86-turn dialogue session, the same model shows Normal Performance (NP) in broad, non-sensitive domains while repeatedly producing Functional Refusal (FR) in provider- or policy-sensitive domains.<n>We operationalize three response regimes (NP, FR, Meta-Narrative; MN) and show that MN role-framing narratives tend to co-occur with refusals in the same sensitive contexts.
  arXiv  Detail & Related papers  (2025-12-15T14:00:15Z)
• InFerActive: Towards Scalable Human Evaluation of Large Language Models through Interactive Inference [14.903507875179033]
  We present InFerActive, an interactive inference system for scalable human evaluation.<n>We demonstrate that InFerActive significantly improves evaluation efficiency and enables more comprehensive assessment of model behavior.
  arXiv  Detail & Related papers  (2025-12-11T02:41:14Z)
• DEBATE: A Large-Scale Benchmark for Role-Playing LLM Agents in Multi-Agent, Long-Form Debates [10.609797175227644]
  We introduce DEBATE, the first large-scale empirical benchmark to evaluate the authenticity of the interaction between multi-agent role-playing LLMs.<n>We systematically evaluate and identify critical discrepancies between simulated and authentic group dynamics.
  arXiv  Detail & Related papers  (2025-10-29T02:21:10Z)
• A Unified Evaluation Framework for Multi-Annotator Tendency Learning [6.801084054135531]
  We propose the first unified evaluation framework with two novel metrics.<n> Difference of Inter-annotator Consistency (DIC) quantifies how well models capture annotator tendencies.<n> Behavior Alignment Explainability (BAE) evaluates how well model explanations reflect annotator behavior and decision relevance.
  arXiv  Detail & Related papers  (2025-08-14T06:50:20Z)
• Evaluating Generalization and Representation Stability in Small LMs via Prompting, Fine-Tuning and Out-of-Distribution Prompts [2.377892000761193]
  We investigate the generalization capabilities of small language models under two popular adaptation paradigms: few-shot prompting and supervised fine-tuning.<n>Our findings highlight critical differences in how small models internalize and generalize knowledge under different adaptation strategies.
  arXiv  Detail & Related papers  (2025-06-16T01:44:26Z)
• InverseScope: Scalable Activation Inversion for Interpreting Large Language Models [5.670123459649656]
  InverseScope is an assumption-light and scalable framework for interpreting neural activations via input inversion.<n>To address the inefficiency of sampling in high-dimensional spaces, we propose a novel conditional generation architecture.<n>We also introduce a quantitative evaluation protocol that tests interpretability hypotheses using the feature consistency rate computed over the sampled inputs.
  arXiv  Detail & Related papers  (2025-06-09T03:59:28Z)
• Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation [57.380464382910375]
  We show that the choice of feedback protocol for evaluation can significantly affect evaluation reliability and induce systematic biases.<n>We find that generator models can flip preferences by embedding distractor features.<n>We offer recommendations for choosing feedback protocols based on dataset characteristics and evaluation objectives.
  arXiv  Detail & Related papers  (2025-04-20T19:05:59Z)
• LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks [106.09361690937618]
  There is an increasing trend towards evaluating NLP models with LLMs instead of human judgments.<n>We provide JUDGE-BENCH, a collection of 20 NLP datasets with human annotations covering a broad range of evaluated properties and types of data.<n>We evaluate 11 current LLMs, covering both open-weight and proprietary models, for their ability to replicate the annotations.
  arXiv  Detail & Related papers  (2024-06-26T14:56:13Z)
• VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models [57.43276586087863]
  Large Vision-Language Models (LVLMs) suffer from hallucination issues, wherein the models generate plausible-sounding but factually incorrect outputs. Existing benchmarks are often limited in scope, focusing mainly on object hallucinations. We introduce a multi-dimensional benchmark covering objects, attributes, and relations, with challenging images selected based on associative biases.
  arXiv  Detail & Related papers  (2024-04-22T04:49:22Z)
• AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents [65.16893197330589]
  Large Language Models (LLMs) have demonstrated their ability to replicate human behaviors across a wide range of scenarios. However, their capability in handling complex, multi-character social interactions has yet to be fully explored. We introduce the Multi-Agent Interaction Evaluation Framework (AntEval), encompassing a novel interaction framework and evaluation methods.
  arXiv  Detail & Related papers  (2024-01-12T11:18:00Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Stateful Offline Contextual Policy Evaluation and Learning [88.9134799076718]
  We study off-policy evaluation and learning from sequential data. We formalize the relevant causal structure of problems such as dynamic personalized pricing. We show improved out-of-sample policy performance in this class of relevant problems.
  arXiv  Detail & Related papers  (2021-10-19T16:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.