ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints

• URL: http://arxiv.org/abs/2202.11523v1
• Date: Wed, 23 Feb 2022 13:57:25 GMT
• Title: ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints
• Authors: Yue Jiang, Wolfgang Stuerzlinger, Christof Lutteroth
• Abstract summary: ReverseORC is a novel reverse engineering (RE) approach to discover diverse layout types and their dynamic resizing behaviours. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. It can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.
• Score: 47.164878414034234
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reverse engineering (RE) of user interfaces (UIs) plays an important role in software evolution. However, the large diversity of UI technologies and the need for UIs to be resizable make this challenging. We propose ReverseORC, a novel RE approach able to discover diverse layout types and their dynamic resizing behaviours independently of their implementation, and to specify them by using OR constraints. Unlike previous RE approaches, ReverseORC infers flexible layout constraint specifications by sampling UIs at different sizes and analyzing the differences between them. It can create specifications that replicate even some non-standard layout managers with complex dynamic layout behaviours. We demonstrate that ReverseORC works across different platforms with very different layout approaches, e.g., for GUIs as well as for the Web. Furthermore, it can be used to detect and fix problems in legacy UIs, extend UIs with enhanced layout behaviours, and support the creation of flexible UI layouts.

Related papers

• PosterLLaVa: Constructing a Unified Multi-modal Layout Generator with LLM [58.67882997399021]
  Our research introduces a unified framework for automated graphic layout generation. Our data-driven method employs structured text (JSON format) and visual instruction tuning to generate layouts. We conduct extensive experiments and achieved state-of-the-art (SOTA) performance on public multi-modal layout generation benchmarks.
  arXiv  Detail & Related papers  (2024-06-05T03:05:52Z)
• Compositional Generative Inverse Design [69.22782875567547]
  Inverse design, where we seek to design input variables in order to optimize an underlying objective function, is an important problem. We show that by instead optimizing over the learned energy function captured by the diffusion model, we can avoid such adversarial examples. In an N-body interaction task and a challenging 2D multi-airfoil design task, we demonstrate that by composing the learned diffusion model at test time, our method allows us to design initial states and boundary shapes.
  arXiv  Detail & Related papers  (2024-01-24T01:33:39Z)
• GUILGET: GUI Layout GEneration with Transformer [26.457270239234383]
  The goal is to support the initial step of GUI design by producing realistic and diverse GUI layouts. GUILGET is based on transformers in order to capture the semantic in relationships between elements from GUI-AG. Our experiments, which are conducted on the CLAY dataset, reveal that our model has the best understanding of relationships from GUI-AG.
  arXiv  Detail & Related papers  (2023-04-18T14:27:34Z)
• LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer [80.61492265221817]
  Graphic layout designs play an essential role in visual communication. Yet handcrafting layout designs is skill-demanding, time-consuming, and non-scalable to batch production. Generative models emerge to make design automation scalable but it remains non-trivial to produce designs that comply with designers' desires.
  arXiv  Detail & Related papers  (2022-12-19T21:57:35Z)
• LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction [37.6871815321083]
  Conditional graphic layout generation is a challenging task that has not been well-studied yet. We propose a constraint serialization scheme, a sequence-to-sequence transformation, and a decoding space restriction strategy. Experiments demonstrate that LayoutFormer++ outperforms existing approaches on all the tasks in terms of both better generation quality and less constraint violation.
  arXiv  Detail & Related papers  (2022-08-17T02:43:23Z)
• Constrained Graphic Layout Generation via Latent Optimization [17.05026043385661]
  We generate graphic layouts that can flexibly incorporate design semantics, either specified implicitly or explicitly by a user. Our approach builds on a generative layout model based on a Transformer architecture, and formulates the layout generation as a constrained optimization problem. We show in the experiments that our approach is capable of generating realistic layouts in both constrained and unconstrained generation tasks with a single model.
  arXiv  Detail & Related papers  (2021-08-02T13:04:11Z)
• Magic Layouts: Structural Prior for Component Detection in User Interface Designs [28.394160581239174]
  We present Magic Layouts; a method for parsing screenshots or hand-drawn sketches of user interface (UI) layouts. Our core contribution is to extend existing detectors to exploit a learned structural prior for UI designs. We demonstrate within the context an interactive application for rapidly acquiring digital prototypes of user experience (UX) designs.
  arXiv  Detail & Related papers  (2021-06-14T17:20:36Z)
• VINS: Visual Search for Mobile User Interface Design [66.28088601689069]
  This paper introduces VINS, a visual search framework, that takes as input a UI image and retrieves visually similar design examples. The framework achieves a mean Average Precision of 76.39% for the UI detection and high performance in querying similar UI designs.
  arXiv  Detail & Related papers  (2021-02-10T01:46:33Z)
• ORCSolver: An Efficient Solver for Adaptive GUI Layout with OR-Constraints [63.59902335363947]
  ORCr is a novel solving technique for adaptive ORC layouts based on a branch-and-bound approach with preprocessing. We demonstrate that ORCr simplifies ORC specifications at runtime and our approach can solve ORC layout specifications efficiently at near-interactive rates.
  arXiv  Detail & Related papers  (2020-02-23T15:46:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.