Artificial intelligence for context-aware visual change detection in software test automation

• URL: http://arxiv.org/abs/2405.00874v2
• Date: Mon, 06 Oct 2025 20:23:48 GMT
• Title: Artificial intelligence for context-aware visual change detection in software test automation
• Abstract summary: We propose a novel graph-based approach for context-aware visual change detection in software test automation.<n>We evaluate our approach on a curated dataset of real-world software screenshots and demonstrate that it reliably detects both simple and complex UI changes.
• Score: 5.182076496564637
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated software testing is integral to the software development process, streamlining workflows and ensuring product reliability. Visual testing, particularly for user interface (UI) and user experience (UX) validation, plays a vital role in maintaining software quality. However, conventional techniques such as pixel-wise comparison and region-based visual change detection often fail to capture contextual similarities, subtle variations, and spatial relationships between UI elements. In this paper, we propose a novel graph-based approach for context-aware visual change detection in software test automation. Our method leverages a machine learning model (YOLOv5) to detect UI controls from software screenshots and constructs a graph that models their contextual and spatial relationships. This graph structure is then used to identify correspondences between UI elements across software versions and to detect meaningful changes. The proposed method incorporates a recursive similarity computation that combines structural, visual, and textual cues, offering a robust and holistic model of UI changes. We evaluate our approach on a curated dataset of real-world software screenshots and demonstrate that it reliably detects both simple and complex UI changes. Our method significantly outperforms pixel-wise and region-based baselines, especially in scenarios requiring contextual understanding. We also discuss current limitations related to dataset diversity, baseline complexity, and model generalization, and outline planned future improvements. Overall, our work advances the state of the art in visual change detection and provides a practical solution for enhancing the reliability and maintainability of evolving software interfaces.

Related papers

• Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering [17.639753380076417]
  We propose FRAME, a multi-modal, neuro-symbolic embedding technique.<n>It constructs symbolic, graph-based representations of UI components to encode salient relationships.<n> FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy.
  arXiv  Detail & Related papers  (2026-08-09T19:39:02Z)
• AutoFocus: Uncertainty-Aware Active Visual Search for GUI Grounding [8.941979904606308]
  Vision-Language Models (VLMs) have enabled autonomous GUI agents that translate natural language instructions into executable screen coordinates.<n>Existing zoom-in strategies rely on fixed anchors, grids, or reinforcement learning.<n>We propose AutoFocus, a training-free, uncertainty-aware active visual search framework for GUI grounding.
  arXiv  Detail & Related papers  (2026-05-04T14:18:46Z)
• Multi-modal user interface control detection using cross-attention [6.850038413666062]
  We introduce a novel multi-modal extension of YOLOv5 that integrates GPT-generated textual descriptions of UI images into the detection pipeline.<n>We evaluate the proposed framework on a large dataset of over 16,000 annotated UI screenshots spanning 23 control classes.
  arXiv  Detail & Related papers  (2026-04-08T10:46:52Z)
• Computer-Using World Model [58.59112582915026]
  We introduce the Computer-Using World Model (CUWM), a world model for desktop software that predicts the next user interface (UI) state.<n> CUWM first predicts a textual description of agent-relevant state changes, and then realizes these changes visually to synthesize the next screenshot.<n>We evaluate CUWM via test-time action search, where a frozen agent uses the world model to simulate and compare candidate actions before execution.
  arXiv  Detail & Related papers  (2026-02-19T13:48:29Z)
• GEBench: Benchmarking Image Generation Models as GUI Environments [49.513441724802135]
  We introduce GEBench, a benchmark for evaluating dynamic interaction and temporal coherence in GUI generation.<n>GE-Score is a novel five-dimensional metric that assesses Goal Achievement, Interaction Logic, Content Consistency, UI Plausibility, and Visual Quality.<n>Our findings identify icon interpretation, text rendering, and localization precision as critical bottlenecks.
  arXiv  Detail & Related papers  (2026-02-09T18:52:02Z)
• How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing [56.60465182650588]
  We introduce three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning.<n>We propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment.<n>We find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models.
  arXiv  Detail & Related papers  (2026-02-02T09:24:45Z)
• AFRAgent : An Adaptive Feature Renormalization Based High Resolution Aware GUI agent [21.148033135113927]
  We introduce an instruct-BLIP-based multimodal architecture that achieves superior performance in GUI automation.<n>We propose an adaptive feature renormalization-based (a token-level affine transformation) technique that effectively enriches low-resolution image embeddings.<n>We evaluate AFRAgent on Meta-GUI and AITW benchmarks, establishing a new state-of-the-art baseline for smartphone automation.
  arXiv  Detail & Related papers  (2025-11-30T11:32:54Z)
• Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support [1.506501956463029]
  Development of web-based dashboards for risk analysis and decision making often challenged by difficulty in big, multidimensional data.<n>We introduce a generative AI framework that automates the creation of interactive geospatial dashboards from user-defined inputs.
  arXiv  Detail & Related papers  (2025-10-10T10:58:15Z)
• Mano Technical Report [29.551514304095296]
  Mano is a robust GUI agent built upon a multi-modal foundation model pre-trained on extensive web and computer system data.<n>Mano demonstrates state-of-the-art performance on multiple GUI benchmarks, including Mind2Web and OSWorld.
  arXiv  Detail & Related papers  (2025-09-22T03:13:58Z)
• LLMShot: Reducing snapshot testing maintenance via LLMs [0.5218155982819203]
  Snapshot testing has emerged as a critical technique for UI validation in modern software development.<n>This paper introduces LLMShot, a novel framework that leverages Vision-Language Models (VLMs) to automatically analyze snapshot test failures.
  arXiv  Detail & Related papers  (2025-07-14T08:47:19Z)
• Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents [15.303188467166752]
  We present CogniGUI, a cognitive framework developed to overcome limitations by enabling adaptive learning for GUI automation resembling human-like behavior.<n>To assess the generalization and adaptability of agent systems, we introduce ScreenSeek, a comprehensive benchmark that includes multi application navigation, dynamic state transitions, and cross interface coherence.<n> Experimental results demonstrate that CogniGUI surpasses state-of-the-art methods in both the current GUI grounding benchmarks and our newly proposed benchmark.
  arXiv  Detail & Related papers  (2025-06-22T06:30:52Z)
• Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models [45.498315114762484]
  Tests aim to determine unintuitive behavior of the software as it is presented to the end-user.<n>They provide valuable feedback for the development of the software, but are time-intensive to conduct.<n>We present GERALLT, a system that uses Large Language Models (LLMs) to perform exploratory tests of the Graphical User Interface (GUI) of a real-life engineering software.
  arXiv  Detail & Related papers  (2025-05-23T12:53:28Z)
• A Survey on (M)LLM-Based GUI Agents [62.57899977018417]
  Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction. Recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms. This survey identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control.
  arXiv  Detail & Related papers  (2025-03-27T17:58:31Z)
• TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmark [61.412934963260724]
  Existing diffusion-based text-to-image models often struggle to accurately embed text within images. We introduce TextInVision, a large-scale, text and prompt complexity driven benchmark to evaluate the ability of diffusion models to integrate visual text into images.
  arXiv  Detail & Related papers  (2025-03-17T21:36:31Z)
• Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems [57.30711059396246]
  Current Graphical User Interface (GUI) grounding systems locate interface elements based on natural language instructions. Inspired by human dual-system cognition, we present Focus, a novel GUI grounding framework that combines fast prediction with systematic analysis.
  arXiv  Detail & Related papers  (2025-03-09T06:14:17Z)
• GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent [24.97846085313314]
  We propose a formalized and comprehensive environment to evaluate the entire process of automated GUI Testing. We divide the testing process into three key subtasks: test intention generation, test task execution, and GUI defect detection. It evaluates the performance of different models using three data types: real mobile applications, mobile applications with artificially injected defects, and synthetic data.
  arXiv  Detail & Related papers  (2024-12-24T13:41:47Z)
• Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining [67.87810796668981]
  Information-Sensitive Cropping (ISC) and Self-Refining Dual Learning (SRDL) Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations. These improvements translate to significant gains in both web and OS agent downstream tasks.
  arXiv  Detail & Related papers  (2024-12-13T18:40:10Z)
• Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction [69.57190742976091]
  We introduce Aguvis, a unified vision-based framework for autonomous GUI agents. Our approach leverages image-based observations, and grounding instructions in natural language to visual elements. To address the limitations of previous work, we integrate explicit planning and reasoning within the model.
  arXiv  Detail & Related papers  (2024-12-05T18:58:26Z)
• ShowUI: One Vision-Language-Action Model for GUI Visual Agent [80.50062396585004]
  Building Graphical User Interface (GUI) assistants holds significant promise for enhancing human workflow productivity. We develop a vision-language-action model in digital world, namely ShowUI, which features the following innovations. ShowUI, a lightweight 2B model using 256K data, achieves a strong 75.1% accuracy in zero-shot screenshot grounding.
  arXiv  Detail & Related papers  (2024-11-26T14:29:47Z)
• Grounded GUI Understanding for Vision Based Spatial Intelligent Agent: Exemplified by Virtual Reality Apps [41.601579396549404]
  We propose the first zero-shot cOntext-sensitive inteRactable GUI ElemeNT dEtection framework for virtual Reality apps, named Orienter. By imitating human behaviors, Orienter observes and understands the semantic contexts of VR app scenes first, before performing the detection.
  arXiv  Detail & Related papers  (2024-09-17T00:58:00Z)
• Vision-driven Automated Mobile GUI Testing via Multimodal Large Language Model [27.97964877860671]
  This paper proposes a vision-driven automated GUI testing approach to detect non-crash functional bugs with Multimodal Large Language Models. It begins by extracting GUI text information and aligning it with screenshots to form a vision prompt, enabling MLLM to understand GUI context. VisionDroid identifies 29 new bugs on Google Play, of which 19 have been confirmed and fixed.
  arXiv  Detail & Related papers  (2024-07-03T11:58:09Z)
• Towards Unified Multi-granularity Text Detection with Interactive Attention [56.79437272168507]
  "Detect Any Text" is an advanced paradigm that unifies scene text detection, layout analysis, and document page detection into a cohesive, end-to-end model. A pivotal innovation in DAT is the across-granularity interactive attention module, which significantly enhances the representation learning of text instances. Tests demonstrate that DAT achieves state-of-the-art performances across a variety of text-related benchmarks.
  arXiv  Detail & Related papers  (2024-05-30T07:25:23Z)
• FINEMATCH: Aspect-based Fine-grained Image and Text Mismatch Detection and Correction [66.98008357232428]
  We propose FineMatch, a new aspect-based fine-grained text and image matching benchmark. FineMatch focuses on text and image mismatch detection and correction. We show that models trained on FineMatch demonstrate enhanced proficiency in detecting fine-grained text and image mismatches.
  arXiv  Detail & Related papers  (2024-04-23T03:42:14Z)
• Gamified GUI testing with Selenium in the IntelliJ IDE: A Prototype Plugin [0.559239450391449]
  This paper presents GIPGUT: a prototype of a gamification plugin for IntelliJ IDEA. The plugin enhances testers' engagement with typically monotonous and tedious tasks through achievements, rewards, and profile customization. The results indicate high usability and positive reception of the gamification elements.
  arXiv  Detail & Related papers  (2024-03-14T20:11:11Z)
• Automated Testing for Text-to-Image Software [0.0]
  ACTesting is an automated cross-modal testing method for text-to-image (T2I) software. We show that ACTesting can generate error-revealing tests, reducing the text-image consistency by up to 20% compared with the baseline. The results demonstrate that ACTesting can identify abnormal behaviors of T2I software effectively.
  arXiv  Detail & Related papers  (2023-12-20T11:19:23Z)
• Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models [56.257840490146]
  ConCue is a novel approach for improving visual feature extraction in HOI detection. We develop a transformer-based feature extraction module with a multi-tower architecture that integrates contextual cues into both instance and interaction detectors.
  arXiv  Detail & Related papers  (2023-11-26T09:11:32Z)
• Reliable Shot Identification for Complex Event Detection via Visual-Semantic Embedding [72.9370352430965]
  We propose a visual-semantic guided loss method for event detection in videos. Motivated by curriculum learning, we introduce a negative elastic regularization term to start training the classifier with instances of high reliability. An alternative optimization algorithm is developed to solve the proposed challenging non-net regularization problem.
  arXiv  Detail & Related papers  (2021-10-12T11:46:56Z)
• Software Vulnerability Detection via Deep Learning over Disaggregated Code Graph Representation [57.92972327649165]
  This work explores a deep learning approach to automatically learn the insecure patterns from code corpora. Because code naturally admits graph structures with parsing, we develop a novel graph neural network (GNN) to exploit both the semantic context and structural regularity of a program.
  arXiv  Detail & Related papers  (2021-09-07T21:24:36Z)
• Automated User Experience Testing through Multi-Dimensional Performance Impact Analysis [0.0]
  We propose a novel automated user experience testing methodology. It learns how code changes impact the time unit and system tests take, and extrapolates user experience changes based on this information. Our open-source tool achieved 3.7% mean absolute error rate with a random forest regressor.
  arXiv  Detail & Related papers  (2021-04-08T01:18:01Z)
• Applied Awareness: Test-Driven GUI Development using Computer Vision and Cryptography [0.0]
  Test-driven development is impractical: it generally requires an initial implementation of the GUI to generate golden images or to construct interactive test scenarios. We demonstrate a novel and immediately applicable approach of interpreting GUI presentation in terms of backend communications. This focus on backend communication circumvents deficiencies in typical testing methodologies that rely on platform-dependent UI affordances or accessibility features.
  arXiv  Detail & Related papers  (2020-06-05T22:46:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.