DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models

• URL: http://arxiv.org/abs/2411.01606v2
• Date: Thu, 12 Dec 2024 11:27:30 GMT
• Title: DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
• Authors: Mingyue Yuan, Jieshan Chen, Zhenchang Xing, Aaron Quigley, Yuyu Luo, Tianqi Luo, Gelareh Mohammadi, Qinghua Lu, Liming Zhu,
• Abstract summary: DesignRepair is a novel dual-stream design guideline-aware system to examine and repair the design quality issues from both code aspect and rendered page aspect.<n>Our evaluations validated the efficacy and utility of our approach, demonstrating significant enhancements in adherence to design guidelines, accessibility, and user experience metrics.
• Score: 24.54628448382394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rise of Large Language Models (LLMs) has streamlined frontend interface creation through tools like Vercel's V0, yet surfaced challenges in design quality (e.g., accessibility, and usability). Current solutions, often limited by their focus, generalisability, or data dependency, fall short in addressing these complexities. Moreover, none of them examine the quality of LLM-generated UI design. In this work, we introduce DesignRepair, a novel dual-stream design guideline-aware system to examine and repair the UI design quality issues from both code aspect and rendered page aspect. We utilised the mature and popular Material Design as our knowledge base to guide this process. Specifically, we first constructed a comprehensive knowledge base encoding Google's Material Design principles into low-level component knowledge base and high-level system design knowledge base. After that, DesignRepair employs a LLM for the extraction of key components and utilizes the Playwright tool for precise page analysis, aligning these with the established knowledge bases. Finally, we integrate Retrieval-Augmented Generation with state-of-the-art LLMs like GPT-4 to holistically refine and repair frontend code through a strategic divide and conquer approach. Our extensive evaluations validated the efficacy and utility of our approach, demonstrating significant enhancements in adherence to design guidelines, accessibility, and user experience metrics.

Related papers

• CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design [20.69770605071827]
  We introduce CANVAS, a benchmark for VLMs on tool-based user interface design.<n>Our benchmark contains 598 tool-based design tasks paired with ground-truth references sampled from 3.3K mobile UI designs.<n>Results suggest that leading models exhibit more strategic tool invocations, improving design quality.
  arXiv  Detail & Related papers  (2025-11-25T16:13:20Z)
• Generating Software Architecture Description from Source Code using Reverse Engineering and Large Language Model [2.6126272668390373]
  Software Architecture Descriptions (SADs) are essential for managing the inherent complexity of modern software systems.<n>SADs are often missing, outdated, or poorly aligned with the system's actual implementation.<n>We propose a semi-automated generation of SADs from source code by integrating reverse engineering (RE) techniques with a Large Language Model (LLM)
  arXiv  Detail & Related papers  (2025-11-07T11:35:46Z)
• Reinforced Visual Perception with Tools [66.79840157663237]
  We introduce a novel RL algorithm based on GRPO, designed to train models to reason with a suite of four visual tools.<n>We show that our method achieves state-of-the-art performance on several perception-heavy benchmarks.<n>Our ReVPT-3B and ReVPT-7B outperform the instruct models by 9.03% and 9.44% on CV-Bench.
  arXiv  Detail & Related papers  (2025-09-01T17:57:49Z)
• GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking [55.762798168494726]
  GUI-ReRank is a novel framework that integrates rapid embedding-based constrained retrieval models with highly effective MLLM-based reranking techniques.<n>We evaluated our approach on an established NL-based GUI retrieval benchmark.
  arXiv  Detail & Related papers  (2025-08-05T10:17:38Z)
• LOCOFY Large Design Models -- Design to code conversion solution [0.0]
  We introduce the Large Design Models paradigm specifically trained on designs and webpages to enable seamless conversion from design-to-code.<n>We have developed a training and inference pipeline by incorporating data engineering and appropriate model architecture modification.<n>Our models illustrated exceptional end-to-end design-to-code conversion accuracy using a novel preview match score metric.
  arXiv  Detail & Related papers  (2025-07-22T03:54:57Z)
• Computer-Aided Layout Generation for Building Design: A Review [18.703604111298695]
  In this paper, we conduct a review of three major research topics of architecture layout design and generation. For each topic, we present an overview of the leading paradigms, categorized either by research domains (architecture or machine learning) or by user input conditions or constraints.
  arXiv  Detail & Related papers  (2025-04-13T19:00:53Z)
• AesthetiQ: Enhancing Graphic Layout Design via Aesthetic-Aware Preference Alignment of Multi-modal Large Language Models [15.483561230992768]
  Aesthetic-Aware Preference Alignment(AAPA) is a novel technique to train a Multi-modal Large Language Model (MLLM) for layout interfaces. We propose a data filtering protocol utilizing our layout-quality prediction protocol to ensure training happens on high-quality layouts. We demonstrate the efficacy of our approach on two challenging benchmarks - Crello and Webui, showcasing 17%, and 16 improvement over current State-of-The-Art methods.
  arXiv  Detail & Related papers  (2025-03-01T19:05:02Z)
• FlairGPT: Repurposing LLMs for Interior Designs [26.07841568311428]
  We investigate if large language models (LLMs) can be directly utilized for interior design. By systematically probing LLMs, we can reliably generate a list of objects along with relevant constraints. We translate this information into a design layout graph, which is then solved using an off-the-shelf constrained optimization setup.
  arXiv  Detail & Related papers  (2025-01-08T18:01:49Z)
• GLDesigner: Leveraging Multi-Modal LLMs as Designer for Enhanced Aesthetic Text Glyph Layouts [53.568057283934714]
  We propose a VLM-based framework that generates content-aware text logo layouts. We introduce two model techniques to reduce the computation for processing multiple glyph images simultaneously. To support instruction-tuning of out model, we construct two extensive text logo datasets, which are 5x more larger than the existing public dataset.
  arXiv  Detail & Related papers  (2024-11-18T10:04:10Z)
• DesignProbe: A Graphic Design Benchmark for Multimodal Large Language Models [35.10231741092462]
  A well-executed graphic design typically achieves harmony in two levels, from the fine-grained design elements (color, font and layout) to the overall design. With the rapid development of Multimodal Large Language Models (MLLMs), we establish the DesignProbe, a benchmark to investigate the capability of MLLMs in design.
  arXiv  Detail & Related papers  (2024-04-23T07:31:19Z)
• I-Design: Personalized LLM Interior Designer [57.00412237555167]
  I-Design is a personalized interior designer that allows users to generate and visualize their design goals through natural language communication. I-Design starts with a team of large language model agents that engage in dialogues and logical reasoning with one another. The final design is then constructed in 3D by retrieving and integrating assets from an existing object database.
  arXiv  Detail & Related papers  (2024-04-03T16:17:53Z)
• From Concept to Manufacturing: Evaluating Vision-Language Models for Engineering Design [5.268919870502001]
  This paper presents a comprehensive evaluation of vision-language models (VLMs) across a spectrum of engineering design tasks. Specifically in this paper, we assess the capabilities of two VLMs, GPT-4V and LLaVA 1.6 34B, in design tasks such as sketch similarity analysis, CAD generation, topology optimization, manufacturability assessment, and engineering textbook problems.
  arXiv  Detail & Related papers  (2023-11-21T15:20:48Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)
• LayoutNUWA: Revealing the Hidden Layout Expertise of Large Language Models [84.16541551923221]
  We propose a model that treats layout generation as a code generation task to enhance semantic information. We develop a Code Instruct Tuning (CIT) approach comprising three interconnected modules. We attain significant state-of-the-art performance on multiple datasets.
  arXiv  Detail & Related papers  (2023-09-18T06:35:10Z)
• CodeTF: One-stop Transformer Library for State-of-the-art Code LLM [72.1638273937025]
  We present CodeTF, an open-source Transformer-based library for state-of-the-art Code LLMs and code intelligence. Our library supports a collection of pretrained Code LLM models and popular code benchmarks. We hope CodeTF is able to bridge the gap between machine learning/generative AI and software engineering.
  arXiv  Detail & Related papers  (2023-05-31T05:24:48Z)
• Conceptual Design Generation Using Large Language Models [0.34998703934432673]
  Large Language Models (LLMs) can generate seemingly creative outputs from textual prompts. In this paper, we leverage LLMs to generate solutions for a set of 12 design problems and compare them to a baseline of crowdsourced solutions. Expert evaluations indicate that the LLM-generated solutions have higher average feasibility and usefulness. We experiment with prompt engineering and find that leveraging few-shot learning can lead to the generation of solutions that are more similar to the crowdsourced solutions.
  arXiv  Detail & Related papers  (2023-05-30T19:32:39Z)
• LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer [80.61492265221817]
  Graphic layout designs play an essential role in visual communication. Yet handcrafting layout designs is skill-demanding, time-consuming, and non-scalable to batch production. Generative models emerge to make design automation scalable but it remains non-trivial to produce designs that comply with designers' desires.
  arXiv  Detail & Related papers  (2022-12-19T21:57:35Z)
• Designing Machine Learning Toolboxes: Concepts, Principles and Patterns [0.0]
  We provide an overview of key patterns in the design of AI modeling toolboxes. Our analysis can not only explain the design of existing toolboxes, but also guide the development of new ones.
  arXiv  Detail & Related papers  (2021-01-13T08:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.