Improving User Interface Generation Models from Designer Feedback

• URL: http://arxiv.org/abs/2509.16779v1
• Date: Sat, 20 Sep 2025 19:02:53 GMT
• Title: Improving User Interface Generation Models from Designer Feedback
• Abstract summary: Designer feedback is essential to improving performance on UI generation.<n>We find that existing methods based on ratings or rankings are not well-aligned with designers' rationales.
• Score: 9.027073356362182
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite being trained on vast amounts of data, most LLMs are unable to reliably generate well-designed UIs. Designer feedback is essential to improving performance on UI generation; however, we find that existing RLHF methods based on ratings or rankings are not well-aligned with designers' workflows and ignore the rich rationale used to critique and improve UI designs. In this paper, we investigate several approaches for designers to give feedback to UI generation models, using familiar interactions such as commenting, sketching and direct manipulation. We first perform a study with 21 designers where they gave feedback using these interactions, which resulted in ~1500 design annotations. We then use this data to finetune a series of LLMs to generate higher quality UIs. Finally, we evaluate these models with human judges, and we find that our designer-aligned approaches outperform models trained with traditional ranking feedback and all tested baselines, including GPT-5.

Related papers

• Efficient Personalization of Generative User Interfaces [17.40636742119123]
  We develop a sample-efficient personalization method that represents a new user in terms of prior designers.<n>In a technical evaluation, our preference model outperforms both a pretrained UI evaluator and a larger multimodal model.<n>Our findings suggest that lightweight preference elicitation can serve as a practical foundation for personalized generative UI systems.
  arXiv  Detail & Related papers  (2026-04-10T20:05:51Z)
• LikeThis! Empowering App Users to Submit UI Improvement Suggestions Instead of Complaints [6.85181982998051]
  We propose LikeThis!, a GenAI-based approach that takes a user comment with the corresponding screenshot to immediately generate multiple improvement alternatives.<n>The results show that our approach helps generate better feedback from both user and developer perspectives.
  arXiv  Detail & Related papers  (2026-03-04T16:33:35Z)
• DesignPref: Capturing Personal Preferences in Visual Design Generation [19.015205942741837]
  We introduce DesignPref, a dataset of 12k pairwise comparisons of user interface (UI) design generation annotated by 20 professional designers.<n>We show that personalized models consistently outperform aggregated baseline models in predicting individual designers' preferences.<n>Our work provides the first dataset to study personalized visual design evaluation and support future research into modeling individual design taste.
  arXiv  Detail & Related papers  (2025-11-25T17:19:10Z)
• MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces [97.62557395494962]
  We use crowdsourcing to benchmark GPT-4o, Claude, and Llama across 30 interfaces.<n>Our results show that MLLMs approximate human preferences on some dimensions but diverge on others.
  arXiv  Detail & Related papers  (2025-10-09T20:00:41Z)
• Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping [55.98643055756135]
  We introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. We analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs. A user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception.
  arXiv  Detail & Related papers  (2024-10-21T17:39:49Z)
• UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset [10.427243347670965]
  We present a targeted dataset of 3,059 design critiques and quality ratings for 983 mobile UIs. We apply this dataset to achieve 55% performance gain in LLM-generated UI feedback. We discuss future applications of this dataset, including training a reward model for generative UI techniques.
  arXiv  Detail & Related papers  (2024-07-11T20:18:19Z)
• Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation [67.88747330066049]
  Fine-grained feedback captures nuanced distinctions in image quality and prompt-alignment. We show that demonstrating its superiority to coarse-grained feedback is not automatic. We identify key challenges in eliciting and utilizing fine-grained feedback.
  arXiv  Detail & Related papers  (2024-06-24T17:19:34Z)
• Leveraging Human Revisions for Improving Text-to-Layout Models [16.617352120973806]
  We propose using nuanced feedback through the form of human revisions for stronger alignment. Our method, Revision-Aware Reward Models, allows a generative text-to- text model to produce more modern, designer-aligned layouts.
  arXiv  Detail & Related papers  (2024-05-16T01:33:09Z)
• UIClip: A Data-driven Model for Assessing User Interface Design [20.66914084220734]
  We develop a machine-learned model, UIClip, for assessing the design quality and visual relevance of a user interface. We show how UIClip can facilitate downstream applications that rely on instantaneous assessment of UI design quality.
  arXiv  Detail & Related papers  (2024-04-18T20:43:08Z)
• Personalized Language Modeling from Personalized Human Feedback [45.16986573937782]
  Personalized large language models (LLMs) are designed to tailor responses to individual user preferences.<n>We propose Personalized-RLHF, an efficient framework that utilizes a lightweight user model to capture individual user preferences.<n>We show that personalized LLMs trained using P-RLHF generate responses that are more closely aligned with individual user preferences.
  arXiv  Detail & Related papers  (2024-02-06T04:18:58Z)
• Constructive Large Language Models Alignment with Diverse Feedback [76.9578950893839]
  We introduce Constructive and Diverse Feedback (CDF) as a novel method to enhance large language models alignment. We exploit critique feedback for easy problems, refinement feedback for medium problems, and preference feedback for hard problems. By training our model with this diversified feedback, we achieve enhanced alignment performance while using less training data.
  arXiv  Detail & Related papers  (2023-10-10T09:20:14Z)
• UltraFeedback: Boosting Language Models with Scaled AI Feedback [99.4633351133207]
  We present textscUltraFeedback, a large-scale, high-quality, and diversified AI feedback dataset. Our work validates the effectiveness of scaled AI feedback data in constructing strong open-source chat language models.
  arXiv  Detail & Related papers  (2023-10-02T17:40:01Z)
• System-Level Natural Language Feedback [83.24259100437965]
  We show how to use feedback to formalize system-level design decisions in a human-in-the-loop-process. We conduct two case studies of this approach for improving search query and dialog response generation. We show the combination of system-level and instance-level feedback brings further gains.
  arXiv  Detail & Related papers  (2023-06-23T16:21:40Z)
• Chain of Hindsight Aligns Language Models with Feedback [62.68665658130472]
  We propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors.
  arXiv  Detail & Related papers  (2023-02-06T10:28:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.