Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation

• URL: http://arxiv.org/abs/2607.28439v2
• Date: Fri, 31 Jul 2026 05:50:45 GMT
• Title: Beyond a Single Judge: The Evidence-Grounded, Social-Weighted Persona Panel for Generative UI Evaluation
• Abstract summary: GenUI lets large language models synthesize a complete, renderable interface directly from a natural-language instruction.<n>Human evaluation is costly and rater-variant, while LLM-as-a-judge is scalable but reflects only a single implicit viewpoint.<n>We propose the Evidence-Grounded, Social-Weighted Persona Panel (ESPP), a three-stage GenUI evaluation method.
• Score: 19.389121016779573
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative UI (GenUI) lets large language models synthesize a complete, renderable interface directly from a natural-language instruction, but evaluating the quality of what they generate remains an open problem. Human evaluation is costly and rater-variant, while LLM-as-a-judge is scalable but reflects only a single implicit viewpoint, unable to capture how different populations of real users actually perceive the same interface. We propose the Evidence-Grounded, Social-Weighted Persona Panel (ESPP), a three-stage GenUI evaluation method in which a panel of psychologically diverse, evidence-grounded personas independently rates a screenshot, exchanges opinions under a trait-derived, semantically-gated bounded-confidence mechanism, and is aggregated via Delphi-inspired social weighting into a single judgment. ESPP tracks human judgment substantially more closely than a naive single-pass judge, raising Pearson $r$ from $0.716$ to $0.922$, and a prompt-ensemble control recovers only about a third of this gap, isolating genuine persona and evidence grounding as the dominant source of improvement. Beyond this fidelity gain, retaining each panelist's individual rating further reveals that user subgroups agree on overall model rankings yet diverge sharply on specific rating dimensions, a structural disagreement a single homogeneous judge would systematically erase. The codes are available at https://github.com/Wuzheng02/ESPP.

Related papers

• Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging [15.994164469249727]
  EYT-Bench is a human-centered benchmark for evaluating large language models.<n> Personas are sampled from public human-curated corpora.<n>State-of-the-art closed and open-source models are statistically close on subjective dimensions.
  arXiv  Detail & Related papers  (2026-07-11T18:22:49Z)
• PA-User: Simulating Trust and Verification under AI-Generated Content [1.1105673928718571]
  We propose PA-User, a user simulator with three new components.<n>A trust component holds a separate Beta belief over the factuality of each source class.<n>A decision rule picks accept, provenance, or discard for each result.
  arXiv  Detail & Related papers  (2026-06-22T00:45:06Z)
• From Self to Other: Evaluating Demographic Perspective-Taking in LLM Hate Speech Annotation [5.762370982168011]
  Persona-conditioned Large Language Models (models prompted to adopt a specific demographic identity) have been proposed as a way to simulate diverse perspectives at scale.<n>We evaluate three aspects of human social judgement: (i) whether personas from different groups disagree in human-like ways (inter-group disagreement), (ii) whether they become more sensitive when content targets their own identity (in-group sensitivity), and (iii) whether they can accurately predict how another group would react (vicarious prediction).<n>Our results show that no model consistently captures all three dimensions, and performance is highly model-dependent and does not emerge reliably from
  arXiv  Detail & Related papers  (2026-06-04T15:09:58Z)
• PICon: A Multi-Turn Interrogation Framework for Evaluating Persona Agent Consistency [13.864376043056387]
  Large language model (LLM)-based persona agents are rapidly being adopted as scalable proxies for human participants.<n>We propose PICon, an evaluation framework that probes persona agents through logically chained multi-turn questioning.
  arXiv  Detail & Related papers  (2026-03-26T16:34:34Z)
• P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling [66.55381105691818]
  We propose P-GenRM, the first Personalized Generative Reward Model with test-time user-based scaling.<n>P-GenRM transforms preference signals into structured evaluation chains that derive adaptive personas and scoring rubrics.<n>It further clusters users into User Prototypes and introduces a dual-granularity scaling mechanism.
  arXiv  Detail & Related papers  (2026-02-12T16:07:22Z)
• Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) [90.45301024940329]
  Language models (LMs) often struggle to generate diverse, human-like creative content.<n>We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries.<n>We present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect.
  arXiv  Detail & Related papers  (2025-10-27T03:16:21Z)
• MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces [97.62557395494962]
  We use crowdsourcing to benchmark GPT-4o, Claude, and Llama across 30 interfaces.<n>Our results show that MLLMs approximate human preferences on some dimensions but diverge on others.
  arXiv  Detail & Related papers  (2025-10-09T20:00:41Z)
• DisCo: Reinforcement with Diversity Constraints for Multi-Human Generation [60.741022906593685]
  DisCo is the first RL-based framework to directly optimize identity diversity in multi-human generation.<n>DisCo fine-tunes flow-matching models via Group-Relative Policy Optimization.<n>On the DiverseHumans Testset, DisCo achieves 98.6 Unique Face Accuracy and near-perfect Global Identity Spread.
  arXiv  Detail & Related papers  (2025-10-01T19:28:51Z)
• PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations? [0.47745223151611654]
  We propose PerFairX to quantify the trade-offs between personalization and demographic equity in LLM-generated recommendations.<n>We benchmark two state-of-the-art LLMs, ChatGPT and DeepSeek, on movie (MovieLens 10M) and music (Last.fm 360K) datasets.<n>DeepSeek achieves stronger psychological fit but exhibits higher sensitivity to prompt variations, while ChatGPT delivers stable yet less personalized outputs.
  arXiv  Detail & Related papers  (2025-08-20T09:41:53Z)
• Democratizing Reward Design for Personal and Representative Value-Alignment [10.1630183955549]
  We introduce Interactive-Reflective Dialogue Alignment, a method that iteratively engages users in reflecting on and specifying their subjective value definitions. This system learns individual value definitions through language-model-based preference elicitation and constructs personalized reward models. Our findings demonstrate diverse definitions of value-aligned behaviour and show that our system can accurately capture each person's unique understanding.
  arXiv  Detail & Related papers  (2024-10-29T16:37:01Z)
• Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems [103.416202777731]
  We study "persona biases", which we define to be the sensitivity of dialogue models' harmful behaviors contingent upon the personas they adopt. We categorize persona biases into biases in harmful expression and harmful agreement, and establish a comprehensive evaluation framework to measure persona biases in five aspects: Offensiveness, Toxic Continuation, Regard, Stereotype Agreement, and Toxic Agreement.
  arXiv  Detail & Related papers  (2023-10-08T21:03:18Z)
• AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild [77.43884383743872]
  We present AdaFuse, an adaptive multiview fusion method to enhance the features in occluded views. We extensively evaluate the approach on three public datasets including Human3.6M, Total Capture and CMU Panoptic. We also create a large scale synthetic dataset Occlusion-Person, which allows us to perform numerical evaluation on the occluded joints.
  arXiv  Detail & Related papers  (2020-10-26T03:19:46Z)
• Face Recognition: Too Bias, or Not Too Bias? [45.404162391012726]
  We reveal critical insights into problems of bias in state-of-the-art facial recognition systems. We show variations in the optimal scoring threshold for face-pairs across different subgroups. We also do a human evaluation to measure the bias in humans, which supports the hypothesis that such bias exists in human perception.
  arXiv  Detail & Related papers  (2020-02-16T01:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.