Controllable GUI Exploration

• URL: http://arxiv.org/abs/2502.03330v1
• Date: Wed, 05 Feb 2025 16:25:35 GMT
• Title: Controllable GUI Exploration
• Authors: Aryan Garg, Yue Jiang, Antti Oulasvirta,
• Abstract summary: Design tools often fail because they insist on specifying more details than necessary. Recent advances in generative AI have raised hopes of solving this issue but they fail because expressing loose ideas in a prompt is impractical. We propose a diffusion-based approach to the low-effort generation of sketches.
• Score: 27.000567061579726
• License:
• Abstract: During the early stages of interface design, designers need to produce multiple sketches to explore a design space. Design tools often fail to support this critical stage, because they insist on specifying more details than necessary. Although recent advances in generative AI have raised hopes of solving this issue, in practice they fail because expressing loose ideas in a prompt is impractical. In this paper, we propose a diffusion-based approach to the low-effort generation of interface sketches. It breaks new ground by allowing flexible control of the generation process via three types of inputs: A) prompts, B) wireframes, and C) visual flows. The designer can provide any combination of these as input at any level of detail, and will get a diverse gallery of low-fidelity solutions in response. The unique benefit is that large design spaces can be explored rapidly with very little effort in input-specification. We present qualitative results for various combinations of input specifications. Additionally, we demonstrate that our model aligns more accurately with these specifications than other models.

Related papers

• T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation [56.054622766743414]
  We propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation. It enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances. Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models.
  arXiv  Detail & Related papers  (2024-12-18T04:01:32Z)
• DiffDesign: Controllable Diffusion with Meta Prior for Efficient Interior Design Generation [25.532400438564334]
  We propose DiffDesign, a controllable diffusion model with meta priors for efficient interior design generation. Specifically, we utilize the generative priors of a 2D diffusion model pre-trained on a large image dataset as our rendering backbone. We further guide the denoising process by disentangling cross-attention control over design attributes, such as appearance, pose, and size, and introduce an optimal transfer-based alignment module to enforce view consistency.
  arXiv  Detail & Related papers  (2024-11-25T11:36:34Z)
• Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping [55.98643055756135]
  We introduce Sketch2Code, a benchmark that evaluates state-of-the-art Vision Language Models (VLMs) on automating the conversion of rudimentary sketches into webpage prototypes. We analyze ten commercial and open-source models, showing that Sketch2Code is challenging for existing VLMs. A user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception.
  arXiv  Detail & Related papers  (2024-10-21T17:39:49Z)
• CAD-Prompted Generative Models: A Pathway to Feasible and Novel Engineering Designs [4.806185947218336]
  This paper introduces a method that improves the design feasibility by prompting the generation with feasible CAD images. Results demonstrate that the CAD image prompting successfully helps text-to-image models like Stable Diffusion 2.1 create visibly more feasible design images.
  arXiv  Detail & Related papers  (2024-07-11T17:07:32Z)
• DiCTI: Diffusion-based Clothing Designer via Text-guided Input [5.275658744475251]
  DiCTI (Diffusion-based Clothing Designer via Text-guided Input) allows designers to quickly visualize fashion-related ideas using text inputs only. By leveraging a powerful diffusion-based inpainting model conditioned on text inputs, DiCTI is able to synthesize convincing, high-quality images with varied clothing designs.
  arXiv  Detail & Related papers  (2024-07-04T12:48:36Z)
• PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM [58.67882997399021]
  Our research introduces a unified framework for automated graphic layout generation. Our data-driven method employs structured text (JSON format) and visual instruction tuning to generate layouts. We develop an automated text-to-poster system that generates editable posters based on users' design intentions.
  arXiv  Detail & Related papers  (2024-06-05T03:05:52Z)
• Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints [53.66698106829144]
  We propose a unified model to handle a broad range of layout generation tasks. The model is based on continuous diffusion models. Experiment results show that LACE produces high-quality layouts.
  arXiv  Detail & Related papers  (2024-02-07T11:12:41Z)
• Compositional Generative Inverse Design [69.22782875567547]
  Inverse design, where we seek to design input variables in order to optimize an underlying objective function, is an important problem. We show that by instead optimizing over the learned energy function captured by the diffusion model, we can avoid such adversarial examples. In an N-body interaction task and a challenging 2D multi-airfoil design task, we demonstrate that by composing the learned diffusion model at test time, our method allows us to design initial states and boundary shapes.
  arXiv  Detail & Related papers  (2024-01-24T01:33:39Z)
• LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer [80.61492265221817]
  Graphic layout designs play an essential role in visual communication. Yet handcrafting layout designs is skill-demanding, time-consuming, and non-scalable to batch production. Generative models emerge to make design automation scalable but it remains non-trivial to produce designs that comply with designers' desires.
  arXiv  Detail & Related papers  (2022-12-19T21:57:35Z)
• iPLAN: Interactive and Procedural Layout Planning [13.172253981084403]
  We propose a new human-in-the-loop generative model, iPLAN. It is capable of automatically generating layouts, but also interacting with designers throughout the whole procedure. The results show that iPLAN has high fidelity in producing similar layouts to those from human designers.
  arXiv  Detail & Related papers  (2022-03-27T23:21:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.