Proactive Detection of GUI Defects in Multi-Window Scenarios via Multimodal Reasoning

• URL: http://arxiv.org/abs/2604.19081v1
• Date: Tue, 21 Apr 2026 04:47:09 GMT
• Title: Proactive Detection of GUI Defects in Multi-Window Scenarios via Multimodal Reasoning
• Abstract summary: We propose an end-to-end framework for GUI display defect detection in multi-window mobile scenarios.<n>The framework proactively triggers split-screen, foldable, and window-transition states during app exploration.<n>It uses multimodal large language models with chain-of-thought prompting to detect, localize, and explain display defects.
• Score: 19.41663018956925
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-window mobile scenarios, such as split-screen and foldable modes, make GUI display defects more likely by forcing applications to adapt to changing window sizes and dynamic layout reflow. Existing detection techniques are limited in two ways: they are largely passive, analyzing screenshots only after problematic states have been reached, and they are mainly designed for conventional full-screen interfaces, making them less effective in multi-window settings.We propose an end-to-end framework for GUI display defect detection in multi-window mobile scenarios. The framework proactively triggers split-screen, foldable, and window-transition states during app exploration, uses Set-of-Mark (SoM) to align screenshots with widget-level interface elements, and leverages multimodal large language models with chain-of-thought prompting to detect, localize, and explain display defects. We also construct a benchmark of GUI display defects using 50 real-world Android applications.Experimental results show that multi-window settings substantially increase the exposure of layout-related defects, with text truncation increasing by 184% compared with conventional full-screen settings. At the application level, our method detects 40 defect-prone apps with a false positive rate of 10.00% and a false negative rate of 11.11%, outperforming OwlEye and YOLO-based baselines. At the fine-grained level, it achieves the best F1 score of 87.2% for widget occlusion detection.

Related papers

• Spotting Setting-Related UI Display Bugs in Android Apps [19.918077756884852]
  Android provides a wide range of system settings that allow users to control the runtime behaviors of apps, such as screen rotation and UI display.<n> setting-related bugs occur when developers do not fully align their apps with the extensive range of system settings that users can define.<n>These bugs can commonly affect apps' UI, causing setting-related UI display (SUD) bugs that negatively impact user experience.
  arXiv  Detail & Related papers  (2026-07-05T05:17:42Z)
• WinDeskGround: A Benchmark for Robust GUI Grounding in Complex Multi-Window Desktop Environments [49.98994180610182]
  Multimodal Large Language Models (MLLMs) have revolutionized GUI automation, yet their efficacy is largely established on idealized, single-layer interfaces.<n>This paper identifies a critical reliability gap: state-of-the-art agents face distinct challenges in real-world desktop environments.<n>We introduce WinDeskGround, a novel benchmark and synthesis framework tailored for evaluating GUI grounding robustness.
  arXiv  Detail & Related papers  (2026-05-13T02:48:52Z)
• BAMI: Training-Free Bias Mitigation in GUI Grounding [72.1023567399117]
  We introduce textbfBias-Aware Manipulation Inference (BAMI), which incorporates two key manipulations, coarse-to-fine focus and candidate selection, to effectively mitigate these biases.<n>BAMI significantly enhances the accuracy of various GUI grounding models in a training-free setting.
  arXiv  Detail & Related papers  (2026-05-07T17:59:31Z)
• UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Grounding [55.22866422166757]
  Test-time zoom-in methods apply cropping uniformly across all instances with fixed crop sizes, ignoring whether the model is actually uncertain on each case.<n>We propose textbfUI-Zoomer, a training-free adaptive zoom-in framework that treats both the trigger and scale of zoom-in as a prediction uncertainty problem.
  arXiv  Detail & Related papers  (2026-04-15T17:32:28Z)
• \textsc{GUI-Spotlight}: Adaptive Iterative Focus Refinement for Enhanced GUI Visual Grounding [37.69847052653875]
  We introduce GUI-Spotlight, a model trained for image-grounded reasoning.<n>It iteratively narrows its focus to the relevant region of the screen, thereby substantially improving visual grounding accuracy.<n>On the ScreenSpot-Pro benchmark, GUI-Spotlight trained with only 18.5K training samples achieves 52.8% accuracy.
  arXiv  Detail & Related papers  (2025-10-05T05:15:45Z)
• SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing [13.521180435948791]
  We propose a novel end-to-end framework for GUI perception.<n>Instead of using probability-based discrete modeling, we perform continuous modeling of coordinates.<n>This effectively mitigates the limitations inherent in the discrete output characteristics.
  arXiv  Detail & Related papers  (2025-09-05T08:24:12Z)
• Screencast-Based Analysis of User-Perceived GUI Responsiveness [53.53923672866705]
  tool is a technique that measures GUI responsiveness directly from mobile screencasts.<n>It uses computer vision to detect user interactions and analyzes frame-level visual changes to compute two key metrics.<n>tool has been deployed in an industrial testing pipeline and analyzes thousands of screencasts daily.
  arXiv  Detail & Related papers  (2025-08-02T12:13:50Z)
• GTA1: GUI Test-time Scaling Agent [97.58177633084915]
  Graphical user interface (GUI) agents autonomously complete tasks across platforms (eg, Linux) by sequentially decomposing user instructions into action proposals.<n>This paper investigates the aforementioned challenges with our textbfGUI textbfTest-time Scaling textbfAgent, namely GTA1.
  arXiv  Detail & Related papers  (2025-07-08T08:52:18Z)
• GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs [16.85762480091835]
  We propose GUIPilot, an approach for detecting inconsistencies between the mobile design and their implementations.<n>Our experiments show that GUIPilot can achieve 94.5% precision and 99.6% recall in detecting screen inconsistencies.<n>Our industrial case study on applying GUIPilot on a trading mobile application shows that GUIPilot has detected nine application bugs.
  arXiv  Detail & Related papers  (2025-06-09T03:09:48Z)
• Visual Test-time Scaling for GUI Agent Grounding [61.609126885427386]
  We introduce RegionFocus, a visual test-time scaling approach for Vision Language Model Agents.<n>Our approach dynamically zooms in on relevant regions, reducing background clutter and improving grounding accuracy.<n>We observe significant performance gains of 28+% on Screenspot-pro and 24+% on WebVoyager benchmarks.
  arXiv  Detail & Related papers  (2025-05-01T17:45:59Z)
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents [58.18100825673032]
  This paper introduces UI-TARS, a native GUI agent model that solely perceives the screenshots as input and performs human-like interactions.<n>In the OSWorld benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15 steps, outperforming Claude (22.0 and 14.9 respectively)
  arXiv  Detail & Related papers  (2025-01-21T17:48:10Z)
• OmniParser for Pure Vision Based GUI Agent [37.911094082816504]
  Power multimodal models like GPT-4V as a general agent on multiple operating systems are largely underestimated due to the lack of a robust screen parsing technique. textsc Omni significantly improves GPT-4V's performance on ScreenSpot benchmark. textsc Omni with screenshot only outperforms GPT-4V baselines requiring additional information outside of screenshot.
  arXiv  Detail & Related papers  (2024-08-01T00:00:43Z)
• Learning Multi-view Anomaly Detection with Efficient Adaptive Selection [42.94263165352097]
  Single-view tasks will encounter blind spots from other perspectives, resulting in inaccuracies in sample-level prediction.<n>We introduce the Multi-View Anomaly Detection (MVAD) approach, which learns and integrates features from multi-views.<n>Experiments on the Real-IAD dataset under the multi-class setting validate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-07-16T17:26:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.