Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments

• URL: http://arxiv.org/abs/2605.15734v1
• Date: Fri, 15 May 2026 08:43:26 GMT
• Title: Can We Trust AI-Inferred User States. A Psychometric Framework for Validating the Reliability of Users States Classification by LLMs in Operational Environments
• Abstract summary: This paper empirically tests the assumption that the metrics used for user state assessment are stable and interpretable at the level of individual scores.<n>Analyses include both individual score reliability and aggregated reliability, allowing us to distinguish metrics potentially useful for real-time adaptation.<n>The main contribution of this work is the proposal of a replicable evaluation framework, enabling measurable evaluations of metric applicability.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The use of large language models to assess user states in conversational and adaptive systems is based on the assumption that the metrics used for such assessment are stable and interpretable at the level of individual scores. This paper empirically tests this assumption, focusing on the psychometric reliability of artificial intelligence (AI) measures of user states. This study employed replication evaluation procedures to assess the repeatability of a broad set of metrics across three different bimodal large language models (GPT-4o audio, Gemini 2.0 Flash, Gemini 2.5 Flash). Analyses include both individual score reliability and aggregated reliability, allowing us to distinguish metrics potentially useful for real-time adaptation from those that retain their value only in aggregated analyses. The results demonstrate that metric reliability cannot be considered a default property in interpretive domains. The lack of stability at the level of individual scores precludes the interpretation of such scores as indicators of user state in real-time adaptive systems, even if these metrics demonstrate stability after aggregation. At the same time, the study indicates that individually unstable metrics can retain analytical utility in post-hoc studies, identifying rules governing interactions and their relationships with user experience parameters such as satisfaction, trust, and engagement. The main contribution of this work, besides quantifying the severity of the problem (only 31 of 213 metrics met the criteria), is the proposal of a replicable evaluation framework, enabling measurable evaluations of metric applicability. This approach supports more responsible AI design of adaptive systems, in which the interpretation of results requires explicit validation of reliability and monitoring for violations over time.

Related papers

• The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean [2.822987192839875]
  Agent benchmarks are increasingly used to compare large language models (LLMs) and guide deployment decisions.<n>We identify a double measurement confound: execution-critical decisions are performed by a fixed scaffold instead of the model, while the scorer evaluates outputs using criteria that may not reflect task correctness.<n>We unify these issues within a measurement-theoretic framework that characterizes when benchmark scores can be interpreted as evidence of model capability.
  arXiv  Detail & Related papers  (2026-09-06T21:23:11Z)
• Position: Evaluation Scores Are Perishable Knowledge Claims [0.0]
  We argue that evaluation scores should be treated as claims with three properties: formality, scope, and validity windows.<n>We illustrate the cost of mean aggregation on the public HELM leaderboard.
  arXiv  Detail & Related papers  (2026-07-28T18:50:39Z)
• Conformal Reliability: A New Evaluation Metric for Conditional Generation [62.761166941396844]
  Conditional generative models have recently remarkable success in various applications.<n>We propose a novel evaluation metric called reliability based on prediction set at a pre-specified confidence level.<n>We introduce Conformal ReLiability (CReL), a framework that can (i) construct a prediction set and (ii) accurately optimize the reliability score within the constructed prediction set.
  arXiv  Detail & Related papers  (2026-05-29T03:52:44Z)
• Referential Security as a New Paradigm for AI Evaluations [5.748110232428846]
  We propose referential security as a new paradigm for AI evaluation.<n>This approach reframes model identity as an empirically verifiable property and separates referential stability from the substantive security claims it conditions.
  arXiv  Detail & Related papers  (2026-05-25T10:24:10Z)
• On the Reliability of User-Centric Evaluation of Conversational Recommender Systems [0.9112926574395824]
  We present a large-scale empirical study investigating the reliability of user-centric CRS evaluation on static dialogue transcripts.<n>We collect 1,053 annotations from 124 crowd workers on 200 ReDial dialogues using the 18-dimensional CRS-Que framework.<n>Our results show that utilitarian and outcome-oriented dimensions such as accuracy, usefulness, and satisfaction achieve moderate reliability under aggregation.<n>Many dimensions collapse into a single global quality signal, revealing a strong halo effect in third-party judgments.
  arXiv  Detail & Related papers  (2026-02-19T11:10:11Z)
• CCE: Confidence-Consistency Evaluation for Time Series Anomaly Detection [56.302586730134806]
  We introduce Confidence-Consistency Evaluation (CCE), a novel evaluation metric.<n>CCE simultaneously measures prediction confidence and uncertainty consistency.<n>We also establish RankEval, a benchmark for comparing the ranking capabilities of various metrics.
  arXiv  Detail & Related papers  (2025-09-01T03:38:38Z)
• Skewed Score: A statistical framework to assess autograders [2.9645858732618238]
  "LLM-as-a-judge", or autograders, offer a scalable alternative to human evaluation.<n>They have shown mixed reliability and may exhibit systematic biases.<n>We propose a statistical framework that enables researchers to simultaneously assess their autograders.
  arXiv  Detail & Related papers  (2025-07-04T18:45:10Z)
• Reranking-based Generation for Unbiased Perspective Summarization [10.71668103641552]
  We develop a test set for benchmarking metric reliability using human annotations.<n>We show that traditional metrics underperform compared to language model-based metrics, which prove to be strong evaluators.<n>Our findings aim to contribute to the reliable evaluation and development of perspective summarization methods.
  arXiv  Detail & Related papers  (2025-06-19T00:01:43Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Position: AI Evaluation Should Learn from How We Test Humans [65.36614996495983]
  We argue that psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution to the challenges in today's AI evaluations.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Beyond Robustness: Resilience Verification of Tree-Based Classifiers [7.574509994822738]
  We introduce a new measure called resilience and we focus on its verification. We discuss how resilience can be verified by combining a traditional robustness verification technique with a data-independent stability analysis. Our results show that resilience verification is useful and feasible in practice, yielding a more reliable security assessment of both standard and robust decision tree models.
  arXiv  Detail & Related papers  (2021-12-05T23:07:22Z)
• REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation [63.46331073232526]
  We present an enhancement approach to Reference-based EvAluation Metrics for open-domain dialogue systems. A prediction model is designed to estimate the reliability of the given reference set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric.
  arXiv  Detail & Related papers  (2021-05-30T10:04:13Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.