PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation

• URL: http://arxiv.org/abs/2606.05697v1
• Date: Thu, 04 Jun 2026 04:35:16 GMT
• Title: PerceptUI: LLM Agents as Human-Aligned Synthetic Users for UI/UX Evaluation
• Abstract summary: PerceptUI is a framework for persona-conditioned UI/UX evaluation.<n>It predicts how a specific user would answer interface-related questions and produces natural-language rationales.
• Score: 0.7031557790463293
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: User interface (UI) and user experience (UX) evaluation is central to product development, yet reliable feedback still relies on recruiting human participants or running online A/B tests, making early-stage iteration slow and costly. In light of this, recent work has explored Multimodal Large Language Models as proxy evaluators. However, existing approaches either produce surface-level critiques or a judgment that reflects the model's own biases rather than the genuine response of a particular user. We introduce PerceptUI, a framework for persona-conditioned UI/UX evaluation that predicts how a specific user would answer interface-related questions and produces natural-language rationales. PerceptUI is trained in two stages: (i) contrastive reflection fine-tuning distills teacher-generated rationales by extracting lessons from human decisions, and (ii) a reflective prompt-evolution step from the model's own failure traces. Across multiple domains and datasets, PerceptUI achieves human-level realism, generalizes to unseen questions and personas, and yields population-level response distributions.

Related papers

• Learning Dynamic User Personas from Implicit Interaction Streams via Iterative Refinement [4.027292457517535]
  We propose IRIS, a framework that learns dynamic user personas directly from implicit interaction streams.<n>IRIS produces stable personas and distinguishes individual users while revealing limitations of memory-only approaches.<n>We validate IRIS on anonymized real-world Reddit r/AmItheAsshole (AITA) data, with personas built solely from each author's historical interactions.
  arXiv  Detail & Related papers  (2026-07-29T04:56:14Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• From Feelings to Metrics: Understanding and Formalizing How Users Vibe-Test LLMs [48.36707944399574]
  We study how vibe-testing works in practice and then formalize it to support systematic analysis.<n>In experiments on coding benchmarks, we find that combining personalized prompts and user-aware evaluation can change which model is preferred.
  arXiv  Detail & Related papers  (2026-04-15T17:57:08Z)
• HumanLM: Simulating Users with State Alignment Beats Response Imitation [84.89761487596844]
  We propose a novel training framework, HumanLM, which builds user simulators that accurately reflect real users.<n>HumanLM generates natural-language latent states that align with ground-truth responses through reinforcement learning.<n>For extensive evaluation, we develop Humanual, a comprehensive benchmark for simulating real users based on public data.
  arXiv  Detail & Related papers  (2026-02-07T20:26:28Z)
• MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces [97.62557395494962]
  We use crowdsourcing to benchmark GPT-4o, Claude, and Llama across 30 interfaces.<n>Our results show that MLLMs approximate human preferences on some dimensions but diverge on others.
  arXiv  Detail & Related papers  (2025-10-09T20:00:41Z)
• The Era of Real-World Human Interaction: RL from User Conversations [45.2392745984914]
  We introduce Reinforcement Learning from Human Interaction (RLHI), a paradigm that learns directly from in-the-wild user conversations.<n>We develop two complementary methods: (1) RLHI with User-Guided Rewrites, which revises unsatisfactory model outputs based on users' natural-language follow-up responses, and (2) RLHI with User-Based Rewards, which learns via a reward model conditioned on knowledge of the user's long-term interaction history.
  arXiv  Detail & Related papers  (2025-09-29T17:50:31Z)
• User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal [59.120335322495436]
  We analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs.<n>Second, we study harvesting learning signals from such implicit user feedback.
  arXiv  Detail & Related papers  (2025-07-30T23:33:29Z)
• Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models [54.85405423240165]
  We introduce Interactive Reasoning, an interaction design that visualizes chain-of-thought outputs as a hierarchy of topics.<n>We implement interactive reasoning in Hippo, a prototype for AI-assisted decision making in the face of uncertain trade-offs.
  arXiv  Detail & Related papers  (2025-06-30T10:00:43Z)
• Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding [45.81445929920235]
  We introduce WiserUI-Bench, a novel benchmark for assessing models' multimodal understanding of UI/UX design.<n>It includes 300 diverse real-world UI image pairs, each consisting of two design variants A/B-tested at scale by actual companies.<n>Our benchmark supports two core tasks: (1) selecting the more effective UI/UX design by predicting the A/B test verified winner and (2) assessing how well a model, given the winner, can explain its effectiveness in alignment with expert reasoning.
  arXiv  Detail & Related papers  (2025-05-08T08:00:32Z)
• Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profiles [37.43150003866563]
  We introduce the User Simulator with Implicit Profiles (USP), a framework that infers implicit user profiles from human-machine interactions to simulate personalized and realistic dialogues.<n>USP outperforms strong baselines in terms of authenticity and diversity while maintaining comparable consistency.
  arXiv  Detail & Related papers  (2025-02-26T09:26:54Z)
• User Behavior Simulation with Large Language Model based Agents [116.74368915420065]
  We propose an LLM-based agent framework and design a sandbox environment to simulate real user behaviors. Based on extensive experiments, we find that the simulated behaviors of our method are very close to the ones of real humans.
  arXiv  Detail & Related papers  (2023-06-05T02:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.