Bridging Gulfs in UI Generation through Semantic Guidance

• URL: http://arxiv.org/abs/2601.19171v1
• Date: Tue, 27 Jan 2026 04:01:53 GMT
• Title: Bridging Gulfs in UI Generation through Semantic Guidance
• Authors: Seokhyeon Park, Soohyun Lee, Eugene Choi, Hyunwoo Kim, Minkyu Kweon, Yumin Song, Jinwook Seo,
• Abstract summary: We develop a system that enables users to specify semantics, visualize relationships, and extract how semantics are reflected in generated UIs.<n>A comparative user study suggests that our approach enhances users' perceived control over intent expression, outcome interpretation, and facilitates more predictable, iterative refinement.
• Score: 16.245249868262178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While generative AI enables high-fidelity UI generation from text prompts, users struggle to articulate design intent and evaluate or refine results-creating gulfs of execution and evaluation. To understand the information needed for UI generation, we conducted a thematic analysis of UI prompting guidelines, identifying key design semantics and discovering that they are hierarchical and interdependent. Leveraging these findings, we developed a system that enables users to specify semantics, visualize relationships, and extract how semantics are reflected in generated UIs. By making semantics serve as an intermediate representation between human intent and AI output, our system bridges both gulfs by making requirements explicit and outcomes interpretable. A comparative user study suggests that our approach enhances users' perceived control over intent expression, outcome interpretation, and facilitates more predictable, iterative refinement. Our work demonstrates how explicit semantic representation enables systematic and explainable exploration of design possibilities in AI-driven UI design.

Related papers

• ContextualSHAP : Enhancing SHAP Explanations Through Contextual Language Generation [0.0]
  We propose a Python package that extends SHAP by integrating it with a large language model (LLM)<n>This integration is guided by user-defined parameters to tailor the explanation to both the model context and the user perspective.<n>Results: generated explanations were perceived as more understandable and contextually appropriate compared to visual-only outputs.
  arXiv  Detail & Related papers  (2025-12-08T05:18:15Z)
• Evaluating Node-tree Interfaces for AI Explainability [0.5437050212139087]
  This study evaluates user experiences with two distinct AI interfaces - node-tree interfaces and chatbots.<n>Our node-tree interface visually structures AI-generated responses into hierarchically organized, interactive nodes.<n>Our findings suggest that AI interfaces capable of switching between structured visualizations and conversational formats can significantly enhance transparency and user confidence in AI-powered systems.
  arXiv  Detail & Related papers  (2025-10-07T20:48:08Z)
• Generative Interfaces for Language Models [70.25765232527762]
  We propose a paradigm in which large language models (LLMs) respond to user queries by proactively generating user interfaces (UIs)<n>Our framework leverages structured interface-specific representations and iterative refinements to translate user queries into task-specific UIs.<n>Results show that generative interfaces consistently outperform conversational ones, with up to a 72% improvement in human preference.
  arXiv  Detail & Related papers  (2025-08-26T17:43:20Z)
• ThematicPlane: Bridging Tacit User Intent and Latent Spaces for Image Generation [49.805992099208595]
  We introduce ThematicPlane, a system that enables users to navigate and manipulate high-level semantic concepts.<n>This interface bridges the gap between tacit creative intent and system control.
  arXiv  Detail & Related papers  (2025-08-08T06:57:14Z)
• Exploring Interpretability for Visual Prompt Tuning with Hierarchical Concepts [39.92376420375139]
  We propose the first framework, named Interpretable Visual Prompt Tuning, to explore interpretability for visual prompts.<n>Visual prompts are linked to human-understandable semantic concepts, represented as a set of category-agnostic prototypes.<n>IVPT aggregates features from these regions to generate interpretable prompts, which are structured hierarchically to explain visual prompts at different granularities.
  arXiv  Detail & Related papers  (2025-03-08T06:12:50Z)
• Leveraging Multimodal LLM for Inspirational User Interface Search [12.470067381902972]
  Existing AI-based UI search methods often miss crucial semantics like target users or the mood of apps.<n>We used a multimodal large language model (MLLM) to extract and interpret semantics from mobile UI images.<n>Our approach significantly outperforms existing UI retrieval methods, offering UI designers a more enriched and contextually relevant search experience.
  arXiv  Detail & Related papers  (2025-01-29T17:38:39Z)
• Found in Translation: semantic approaches for enhancing AI interpretability in face verification [0.4222205362654437]
  This study extends previous work by integrating semantic concepts into XAI frameworks to bridge the comprehension gap between model outputs and human understanding.<n>We propose a novel approach combining global and local explanations, using semantic features defined by user-selected facial landmarks.<n>Results indicate that our semantic-based approach, particularly the most detailed set, offers a more nuanced understanding of model decisions than traditional methods.
  arXiv  Detail & Related papers  (2025-01-06T08:34:53Z)
• Survey of User Interface Design and Interaction Techniques in Generative AI Applications [79.55963742878684]
  We aim to create a compendium of different user-interaction patterns that can be used as a reference for designers and developers alike. We also strive to lower the entry barrier for those attempting to learn more about the design of generative AI applications.
  arXiv  Detail & Related papers  (2024-10-28T23:10:06Z)
• Understanding Before Recommendation: Semantic Aspect-Aware Review Exploitation via Large Language Models [53.337728969143086]
  Recommendation systems harness user-item interactions like clicks and reviews to learn their representations. Previous studies improve recommendation accuracy and interpretability by modeling user preferences across various aspects and intents. We introduce a chain-based prompting approach to uncover semantic aspect-aware interactions.
  arXiv  Detail & Related papers  (2023-12-26T15:44:09Z)
• Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis [96.53859361560505]
  We propose a knowledge graph augmented network (KGAN) to incorporate external knowledge with explicitly syntactic and contextual information. KGAN captures the sentiment feature representations from multiple perspectives, i.e., context-, syntax- and knowledge-based. Experiments on three popular ABSA benchmarks demonstrate the effectiveness and robustness of our KGAN.
  arXiv  Detail & Related papers  (2022-01-13T08:25:53Z)
• A Dependency Syntactic Knowledge Augmented Interactive Architecture for End-to-End Aspect-based Sentiment Analysis [73.74885246830611]
  We propose a novel dependency syntactic knowledge augmented interactive architecture with multi-task learning for end-to-end ABSA. This model is capable of fully exploiting the syntactic knowledge (dependency relations and types) by leveraging a well-designed Dependency Relation Embedded Graph Convolutional Network (DreGcn) Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-04T14:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.