From Human-Level AI Tales to AI Leveling Human Scales

• URL: http://arxiv.org/abs/2602.18911v1
• Date: Sat, 21 Feb 2026 17:27:20 GMT
• Title: From Human-Level AI Tales to AI Leveling Human Scales
• Abstract summary: We build on a set of multi-level scales for different capabilities where each level should represent a probability of success of the whole world population.<n>We calibrate each scale for each capability by compiling publicly released human test data.<n>We evaluate the quality of different mappings using group slicing and post-stratification.
• Score: 36.62583289981655
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Comparing AI models to "human level" is often misleading when benchmark scores are incommensurate or human baselines are drawn from a narrow population. To address this, we propose a framework that calibrates items against the 'world population' and report performance on a common, human-anchored scale. Concretely, we build on a set of multi-level scales for different capabilities where each level should represent a probability of success of the whole world population on a logarithmic scale with a base $B$. We calibrate each scale for each capability (reasoning, comprehension, knowledge, volume, etc.) by compiling publicly released human test data spanning education and reasoning benchmarks (PISA, TIMSS, ICAR, UKBioBank, and ReliabilityBench). The base $B$ is estimated by extrapolating between samples with two demographic profiles using LLMs, with the hypothesis that they condense rich information about human populations. We evaluate the quality of different mappings using group slicing and post-stratification. The new techniques allow for the recalibration and standardization of scales relative to the whole-world population.

Related papers

• Mind the Gaps: Mixture-of-Minds for Human Simulation [1.1844977816228046]
  Anacreon is an audience simulation model that targets the individual level within a narrow, well-specified domain.<n>It harvests demographics, psychological traits, and survey responses from public text, and augments each record with a chain-of-emotion.<n>It reduces prompt brittleness by shuffling response options and reduces positive bias by balancing the training distribution.
  arXiv  Detail & Related papers  (2026-08-06T14:48:09Z)
• Computational Turing Test Reveals Systematic Differences Between Human and AI Language [0.0]
  Large language models (LLMs) are increasingly used in the social sciences to simulate human behavior.<n>Existing validation efforts rely heavily on human-judgment-based evaluations.<n>This paper introduces a computational Turing test to assess how closely LLMs approximate human language.
  arXiv  Detail & Related papers  (2025-11-06T08:56:37Z)
• Relative Scaling Laws for LLMs [91.73497548097775]
  Scaling laws describe how language models improve with additional data, parameters, and compute.<n>We introduce relative scaling laws, which track how performance gaps between test distributions evolve with scale.<n>These results show that although scaling improves overall performance, it is not a universal equalizer.
  arXiv  Detail & Related papers  (2025-10-28T16:55:22Z)
• WorldPM: Scaling Human Preference Modeling [130.23230492612214]
  We propose World Preference Modeling$ (WorldPM) to emphasize this scaling potential.<n>We collect preference data from public forums covering diverse user communities.<n>We conduct extensive training using 15M-scale data across models ranging from 1.5B to 72B parameters.
  arXiv  Detail & Related papers  (2025-05-15T17:38:37Z)
• Empirically evaluating commonsense intelligence in large language models with large-scale human judgments [4.212429064310439]
  We propose a method for evaluating common sense in artificial intelligence.<n>We measure the correspondence between a model's judgment and that of a human population.<n>Our framework contributes to the growing call for adapting AI models to human collectivities that possess different, often incompatible, social stocks of knowledge.
  arXiv  Detail & Related papers  (2025-05-15T13:55:27Z)
• Psychometric Alignment: Capturing Human Knowledge Distributions via Language Models [41.324679754114165]
  Language models (LMs) are increasingly used to simulate human-like responses in scenarios where accurately mimicking a population's behavior can guide decision-making. We introduce "psychometric alignment," a metric that measures the extent to which LMs reflect human knowledge distribution. We find significant misalignment between LMs and human populations, though using persona-based prompts can improve alignment.
  arXiv  Detail & Related papers  (2024-07-22T14:02:59Z)
• ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales? [7.307538454513983]
  This study explores the alignment between ChatGPT and human assessments across multiple scales. We sample 300 data instances from three NLE datasets and collect 900 human annotations for both informativeness and clarity scores. Our results show that ChatGPT aligns better with humans in more coarse-grained scales.
  arXiv  Detail & Related papers  (2024-03-26T04:07:08Z)
• HumanBench: Towards General Human-centric Perception with Projector Assisted Pretraining [75.1086193340286]
  It is desirable to have a general pretrain model for versatile human-centric downstream tasks. We propose a textbfHumanBench based on existing datasets to evaluate on the common ground the generalization abilities of different pretraining methods. Our PATH achieves new state-of-the-art results on 17 downstream datasets and on-par results on the other 2 datasets.
  arXiv  Detail & Related papers  (2023-03-10T02:57:07Z)
• Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis [79.18261352971284]
  We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation. We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings. SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no human-annotated training data.
  arXiv  Detail & Related papers  (2022-10-10T22:30:26Z)
• Estimating Structural Disparities for Face Models [54.062512989859265]
  In machine learning, disparity metrics are often defined by measuring the difference in the performance or outcome of a model, across different sub-populations. We explore performing such analysis on computer vision models trained on human faces, and on tasks such as face attribute prediction and affect estimation.
  arXiv  Detail & Related papers  (2022-04-13T05:30:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.