Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework

• URL: http://arxiv.org/abs/2603.04409v1
• Date: Tue, 03 Feb 2026 09:51:47 GMT
• Title: Unpacking Human Preference for LLMs: Demographically Aware Evaluation with the HUMAINE Framework
• Abstract summary: HUMAINE is a framework for multidimensional, demographically aware measurement of human-AI interaction.<n>We evaluate 28 state-of-the-art models across five human-centric dimensions.
• Score: 0.31497178896386996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The evaluation of large language models faces significant challenges. Technical benchmarks often lack real-world relevance, while existing human preference evaluations suffer from unrepresentative sampling, superficial assessment depth, and single-metric reductionism. To address these issues, we introduce HUMAINE, a framework for multidimensional, demographically aware measurement of human-AI interaction. We collected multi-turn, naturalistic conversations from 23,404 participants that were stratified across 22 demographic groups, both in the US and UK, to evaluate 28 state-of-the-art models across five human-centric dimensions. We use a hierarchical Bayesian Bradley-Terry-Davidson (BTD) model, with post-stratification to census data, and our analysis reveals three key insights. \textbf{(1)} We establish a clear performance hierarchy where \texttt{google/gemini-2.5-pro} ranks first overall, with a 95.6\% posterior probability of being the top-ranked model. \textbf{(2)} We uncover significant preference heterogeneity, with user age emerging as the primary demographic axis of disagreement; a model's perceived rank can shift substantially across age groups, exposing failures in generalisation that unrepresentative samples typically mask. \textbf{(3)} We quantify the vast difference in discriminative power across evaluation dimensions, with ambiguous qualities like \textit{Trust, Ethics \& Safety} showing a 65\% tie rate, in stark contrast to the decisive 10\% tie rate for \textit{Overall Winner}. Our work emphasises the need for a more multidimensional, demographically aware perspective in LLM evaluation. We release our complete dataset, interactive leaderboard, and open-source framework.

Related papers

• Sympathetic Framing: Evaluating AI Alignment across Sociodemographic Groups [1.9307952728103126]
  Large Language Models (LLMs) are increasingly shaping how we consume information and form our worldview.<n>This raises concerns beyond bias in AI: do LLMs grasp the emotional nuances conveyed via textual framing?<n>We empirically evaluate how well an array of LLMs aligns with human emotional perception.
  arXiv  Detail & Related papers  (2026-07-22T16:23:26Z)
• Can Persona-Prompted LLMs Emulate Subgroup Values? An Empirical Analysis of Generalisability and Fairness in Cultural Alignment [75.88231994139132]
  We show that even state-of-the-art models like GPT-4.1 achieve only 57.4% accuracy in predicting modal preferences.<n>We find significant pre-existing performance biases, where models better emulate young, male, Chinese, and Christian personas.
  arXiv  Detail & Related papers  (2026-04-14T15:06:13Z)
• Beyond Marginal Distributions: A Framework to Evaluate the Representativeness of Demographic-Aligned LLMs [13.630995219491972]
  We propose a framework for evaluating the representativeness of aligned models.<n>We show the value of our evaluation scheme by comparing two model steering techniques.<n>We conclude that representativeness is a distinct aspect of value alignment.
  arXiv  Detail & Related papers  (2026-01-22T08:45:55Z)
• Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models [118.44328586173556]
  Multimodal Large Language Models (MLLMs) have demonstrated significant advances in visual understanding tasks.<n>Human-MME is a curated benchmark designed to provide a more holistic evaluation of MLLMs in human-centric scene understanding.<n>Our benchmark extends the single-target understanding to the multi-person and multi-image mutual understanding.
  arXiv  Detail & Related papers  (2025-09-30T12:20:57Z)
• AesBiasBench: Evaluating Bias and Alignment in Multimodal Language Models for Personalized Image Aesthetic Assessment [29.2617518199559]
  AesBiasBench is a benchmark designed to evaluate Multimodal Large Language Models (MLLMs)<n>Results indicate that smaller models exhibit stronger stereotype biases, whereas larger models align more closely with human preferences.<n>These findings underscore the importance of identity-aware evaluation frameworks in subjective vision-language tasks.
  arXiv  Detail & Related papers  (2025-09-15T06:25:39Z)
• Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs [65.93003087656754]
  VisFactor is a benchmark that digitizes 20 vision-centric subtests from a well-established cognitive psychology assessment.<n>We evaluate 20 frontier Multimodal Large Language Models (MLLMs) from GPT, Gemini, Claude, LLaMA, Qwen, and SEED families.<n>The best-performing model achieves a score of only 25.19 out of 100, with consistent failures on tasks such as mental rotation, spatial relation inference, and figure-ground discrimination.
  arXiv  Detail & Related papers  (2025-02-23T04:21:32Z)
• VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model [72.13121434085116]
  We introduce VLBiasBench, a benchmark to evaluate biases in Large Vision-Language Models (LVLMs)<n>VLBiasBench features a dataset that covers nine distinct categories of social biases, including age, disability status, gender, nationality, physical appearance, race, religion, profession, social economic status, as well as two intersectional bias categories: race x gender and race x social economic status.<n>We conduct extensive evaluations on 15 open-source models as well as two advanced closed-source models, yielding new insights into the biases present in these models.
  arXiv  Detail & Related papers  (2024-06-20T10:56:59Z)
• Assessment of Multimodal Large Language Models in Alignment with Human Values [43.023052912326314]
  We introduce Ch3Ef, a Compreh3ensive Evaluation dataset and strategy for assessing alignment with human expectations. Ch3Ef dataset contains 1002 human-annotated data samples, covering 12 domains and 46 tasks based on the hhh principle.
  arXiv  Detail & Related papers  (2024-03-26T16:10:21Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation [136.16507050034755]
  Existing human evaluation studies for summarization either exhibit a low inter-annotator agreement or have insufficient scale. We propose a modified summarization salience protocol, Atomic Content Units (ACUs), which is based on fine-grained semantic units. We curate the Robust Summarization Evaluation (RoSE) benchmark, a large human evaluation dataset consisting of 22,000 summary-level annotations over 28 top-performing systems.
  arXiv  Detail & Related papers  (2022-12-15T17:26:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.