Boosting GUI Prototyping with Diffusion Models

• URL: http://arxiv.org/abs/2306.06233v1
• Date: Fri, 9 Jun 2023 20:08:46 GMT
• Title: Boosting GUI Prototyping with Diffusion Models
• Authors: Jialiang Wei, Anne-Lise Courbis, Thomas Lambolais, Binbin Xu, Pierre Louis Bernard, G\'erard Dray
• Abstract summary: Deep learning models such as Stable Diffusion have emerged as a powerful text-to-image tool. We propose UI-Diffuser, an approach that leverages Stable Diffusion to generate mobile UIs. Preliminary results show that UI-Diffuser provides an efficient and cost-effective way to generate mobile GUI designs.
• Score: 0.440401067183266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: GUI (graphical user interface) prototyping is a widely-used technique in requirements engineering for gathering and refining requirements, reducing development risks and increasing stakeholder engagement. However, GUI prototyping can be a time-consuming and costly process. In recent years, deep learning models such as Stable Diffusion have emerged as a powerful text-to-image tool capable of generating detailed images based on text prompts. In this paper, we propose UI-Diffuser, an approach that leverages Stable Diffusion to generate mobile UIs through simple textual descriptions and UI components. Preliminary results show that UI-Diffuser provides an efficient and cost-effective way to generate mobile GUI designs while reducing the need for extensive prototyping efforts. This approach has the potential to significantly improve the speed and efficiency of GUI prototyping in requirements engineering.

Related papers

• Self-Elicitation of Requirements with Automated GUI Prototyping [12.281152349482024]
  SERGUI is a novel approach enabling the Self-Elicitation of Requirements based on an automated GUI prototyping assistant. SerGUI exploits the vast prototyping knowledge embodied in a large-scale GUI repository through Natural Language Requirements (NLR) based GUI retrieval. To measure the effectiveness of our approach, we conducted a preliminary evaluation.
  arXiv  Detail & Related papers  (2024-09-24T18:40:38Z)
• GUICourse: From General Vision Language Models to Versatile GUI Agents [75.5150601913659]
  We contribute GUICourse, a suite of datasets to train visual-based GUI agents. First, we introduce the GUIEnv dataset to strengthen the OCR and grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat datasets to enrich their knowledge of GUI components and interactions.
  arXiv  Detail & Related papers  (2024-06-17T08:30:55Z)
• GUI-WORLD: A Dataset for GUI-oriented Multimodal LLM-based Agents [73.9254861755974]
  This paper introduces a new dataset, called GUI-World, which features meticulously crafted Human-MLLM annotations. We evaluate the capabilities of current state-of-the-art MLLMs, including ImageLLMs and VideoLLMs, in understanding various types of GUI content.
  arXiv  Detail & Related papers  (2024-06-16T06:56:53Z)
• Interlinking User Stories and GUI Prototyping: A Semi-Automatic LLM-based Approach [55.762798168494726]
  We present a novel Large Language Model (LLM)-based approach for validating the implementation of functional NL-based requirements in a graphical user interface (GUI) prototype. Our approach aims to detect functional user stories that are not implemented in a GUI prototype and provides recommendations for suitable GUI components directly implementing the requirements.
  arXiv  Detail & Related papers  (2024-06-12T11:59:26Z)
• Tell Me What's Next: Textual Foresight for Generic UI Representations [65.10591722192609]
  We propose Textual Foresight, a novel pretraining objective for learning UI screen representations. Textual Foresight generates global text descriptions of future UI states given a current UI and local action taken. We train with our newly constructed mobile app dataset, OpenApp, which results in the first public dataset for app UI representation learning.
  arXiv  Detail & Related papers  (2024-06-12T02:43:19Z)
• Graph4GUI: Graph Neural Networks for Representing Graphical User Interfaces [27.84098739594353]
  Graph4GUI exploits graph neural networks to capture individual elements' properties and semantic-visuo-spatial constraints in a layout. The learned representation demonstrated its effectiveness in multiple tasks, especially generating designs in a challenging GUI autocompletion task.
  arXiv  Detail & Related papers  (2024-04-21T04:06:09Z)
• From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces [66.85108822706489]
  This paper focuses on creating agents that interact with the digital world using the same conceptual interface that humans commonly use. It is possible for such agents to outperform human crowdworkers on the MiniWob++ benchmark of GUI-based instruction following tasks.
  arXiv  Detail & Related papers  (2023-05-31T23:39:18Z)
• GUILGET: GUI Layout GEneration with Transformer [26.457270239234383]
  The goal is to support the initial step of GUI design by producing realistic and diverse GUI layouts. GUILGET is based on transformers in order to capture the semantic in relationships between elements from GUI-AG. Our experiments, which are conducted on the CLAY dataset, reveal that our model has the best understanding of relationships from GUI-AG.
  arXiv  Detail & Related papers  (2023-04-18T14:27:34Z)
• VINS: Visual Search for Mobile User Interface Design [66.28088601689069]
  This paper introduces VINS, a visual search framework, that takes as input a UI image and retrieves visually similar design examples. The framework achieves a mean Average Precision of 76.39% for the UI detection and high performance in querying similar UI designs.
  arXiv  Detail & Related papers  (2021-02-10T01:46:33Z)
• GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks [0.0]
  We develop a model GUIGAN to automatically generate GUI designs. Our model significantly outperforms the best of the baseline methods by 30.77% in Frechet Inception distance (FID) and 12.35% in 1-Nearest Neighbor Accuracy (1-NNA)
  arXiv  Detail & Related papers  (2021-01-25T09:42:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.