Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering

• URL: http://arxiv.org/abs/2608.08880v1
• Date: Sun, 09 Aug 2026 19:39:02 GMT
• Title: Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering
• Abstract summary: We propose FRAME, a multi-modal, neuro-symbolic embedding technique.<n>It constructs symbolic, graph-based representations of UI components to encode salient relationships.<n> FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy.
• Score: 17.639753380076417
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User Interface (UI) programming is challenging due to the complex abstraction gap between code and graphical software representations. To bridge this gap, UI programming tools often rely on screen retrieval and clustering, which require accurate similarity measures based on overlapping features. However, computing feature-oriented similarity is difficult because screens with similar functionality often exhibit design variations. To address this, we propose FRAME (ReinForced UseR InterfAce Screen EMbedding with Graphical Structural ComprEhension), a multi-modal, neuro-symbolic embedding technique. FRAME constructs symbolic, graph-based representations of UI components to encode salient relationships and capture feature patterns across different screens. It leverages large vision-language models for visual and lexical encoding, alongside a novel UI-specific computational geometry algorithm that enables weighted embedding propagation. Across three benchmarks, FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy. A comprehensive ablation study further confirms the benefit of each component, demonstrating FRAME's potential for enhancing automated UI design and testing tools.

Related papers

• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments [49.98994180610182]
  Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces.<n>This paper identifies a critical reliability gap: state-of-the-art agents face distinct challenges in real-world desktop environments.<n>We introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness.
  arXiv  Detail & Related papers  (2026-05-13T02:48:52Z)
• Multi-modal user interface control detection using cross-attention [6.850038413666062]
  We introduce a novel multi-modal extension of YOLOv5 that integrates GPT-generated textual descriptions of UI images into the detection pipeline.<n>We evaluate the proposed framework on a large dataset of over 16,000 annotated UI screenshots spanning 23 control classes.
  arXiv  Detail & Related papers  (2026-04-08T10:46:52Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• GEBench: Benchmarking Image Generation Models as GUI Environments [49.513441724802135]
  We introduce GEBench, a benchmark for evaluating dynamic interaction and temporal coherence in GUI generation.<n>GE-Score is a novel five-dimensional metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality.<n>Our findings identify icon interpretation, text rendering, and localization precision as critical bottlenecks.
  arXiv  Detail & Related papers  (2026-02-09T18:52:02Z)
• UISearch: Graph-Based Embeddings for Multimodal Enterprise UI Screenshots Retrieval [1.3563834727527375]
  We present a novel graph-based representation that converts UI screenshots into attributed graphs encoding hierarchical relationships and spatial arrangements.<n>A contrastive graph autoencoder learns embeddings preserving multi-level similarity across visual, structural, and semantic properties.<n>We implement this representation in UISearch, a multi-modal search framework that combines structural embeddings with semantic search through a composable query language.
  arXiv  Detail & Related papers  (2025-11-24T18:20:08Z)
• Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems [57.30711059396246]
  Current Graphical User Interface (GUI) grounding systems locate interface elements based on natural language instructions.<n>Inspired by human dual-system cognition, we present Focus, a novel GUI grounding framework that combines fast prediction with systematic analysis.
  arXiv  Detail & Related papers  (2025-03-09T06:14:17Z)
• Artificial intelligence for context-aware visual change detection in software test automation [5.182076496564637]
  We propose a novel graph-based approach for context-aware visual change detection in software test automation.<n>We evaluate our approach on a curated dataset of real-world software screenshots and demonstrate that it reliably detects both simple and complex UI changes.
  arXiv  Detail & Related papers  (2024-05-01T21:22:33Z)
• Computer User Interface Understanding. A New Dataset and a Learning Framework [2.4473568032515147]
  We introduce the harder task of computer UI understanding. We present a dataset with a set of videos where a user is performing a sequence of actions and each image shows the desktop contents at that time point. We also present a framework that is composed of a synthetic sample generation pipeline to augment the dataset with relevant characteristics.
  arXiv  Detail & Related papers  (2024-03-15T10:26:52Z)
• Multi-task Learning with 3D-Aware Regularization [55.97507478913053]
  We propose a structured 3D-aware regularizer which interfaces multiple tasks through the projection of features extracted from an image encoder to a shared 3D feature space. We show that the proposed method is architecture agnostic and can be plugged into various prior multi-task backbones to improve their performance.
  arXiv  Detail & Related papers  (2023-10-02T08:49:56Z)
• Closing the Loop: Graph Networks to Unify Semantic Objects and Visual Features for Multi-object Scenes [2.236663830879273]
  Loop Closure Detection (LCD) is essential to minimize drift when recognizing previously visited places. Visual Bag-of-Words (vBoW) has been an LCD algorithm of choice for many state-of-the-art SLAM systems. This paper proposes SymbioLCD2, which creates a unified graph structure to integrate semantic objects and visual features symbiotically.
  arXiv  Detail & Related papers  (2022-09-24T00:42:33Z)
• Spatio-Temporal Representation Factorization for Video-based Person Re-Identification [55.01276167336187]
  We propose Spatio-Temporal Representation Factorization module (STRF) for re-ID. STRF is a flexible new computational unit that can be used in conjunction with most existing 3D convolutional neural network architectures for re-ID. We empirically show that STRF improves performance of various existing baseline architectures while demonstrating new state-of-the-art results.
  arXiv  Detail & Related papers  (2021-07-25T19:29:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.