GLIB: Towards Automated Test Oracle for Graphically-Rich Applications

• URL: http://arxiv.org/abs/2106.10507v1
• Date: Sat, 19 Jun 2021 14:50:43 GMT
• Title: GLIB: Towards Automated Test Oracle for Graphically-Rich Applications
• Abstract summary: We propose textttGLIB based on a code-based data augmentation technique to detect game GUI glitches. textttGLIB can achieve 100% precision and 99.5% recall in detecting non-crashing bugs such as game GUI glitches.
• Score: 26.940850357831657
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Graphically-rich applications such as games are ubiquitous with attractive visual effects of Graphical User Interface (GUI) that offers a bridge between software applications and end-users. However, various types of graphical glitches may arise from such GUI complexity and have become one of the main component of software compatibility issues. Our study on bug reports from game development teams in NetEase Inc. indicates that graphical glitches frequently occur during the GUI rendering and severely degrade the quality of graphically-rich applications such as video games. Existing automated testing techniques for such applications focus mainly on generating various GUI test sequences and check whether the test sequences can cause crashes. These techniques require constant human attention to captures non-crashing bugs such as bugs causing graphical glitches. In this paper, we present the first step in automating the test oracle for detecting non-crashing bugs in graphically-rich applications. Specifically, we propose \texttt{GLIB} based on a code-based data augmentation technique to detect game GUI glitches. We perform an evaluation of \texttt{GLIB} on 20 real-world game apps (with bug reports available) and the result shows that \texttt{GLIB} can achieve 100\% precision and 99.5\% recall in detecting non-crashing bugs such as game GUI glitches. Practical application of \texttt{GLIB} on another 14 real-world games (without bug reports) further demonstrates that \texttt{GLIB} can effectively uncover GUI glitches, with 48 of 53 bugs reported by \texttt{GLIB} having been confirmed and fixed so far.

Related papers

• A History-Aware Visually Grounded Critic for Computer Use Agents [70.50900668601969]
  HiViG is a test-time framework built around a multimodal critic trained on real GUI trajectories.<n>At test time, HiViG integrates the critic into the policy decision loop to provide macro-action history.<n>Visually grounded critique verifies raw execution coordinates against the current screenshot to intercept errors before execution.
  arXiv  Detail & Related papers  (2026-06-09T16:39:10Z)
• ViBR: Automated Bug Replay from Video-based Reports using Vision-Language Models [17.094299668194378]
  ViBR is a lightweight and fully automated approach that reproduces bugs directly from GUI recordings.<n>Results show that ViBR successfully reproduces 72% of bug recordings, significantly outperforming state-of-the-art baselines and ablation variants.
  arXiv  Detail & Related papers  (2026-04-21T18:28:02Z)
• PlayCoder: Making LLM-Generated GUI Code Playable [30.115742334937977]
  Existing benchmarks mainly evaluate correctness through test cases, which are inadequate for GUI applications.<n>We introduce PlayEval, a repository-aware benchmark built from 43 multilingual GUI applications in Python, TypeScript, and JavaScript.<n>We also propose Play@k, a metric that measures whether at least one of *k* generated candidates can be played end-to-end without logical errors.
  arXiv  Detail & Related papers  (2026-04-21T17:59:16Z)
• A Hybrid Co-Finetuning Approach for Visual Bug Detection in Video Games [3.5838409897789467]
  We propose a hybrid Co-FineTuning (CFT) method that effectively integrates both labeled and unlabeled data.<n>We show that CFT maintains competitive performance even when trained with only 50% of the labeled data from the target game.
  arXiv  Detail & Related papers  (2025-10-04T00:43:10Z)
• FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games [56.81554611870848]
  We introduce FlashAdventure, a benchmark of 34 Flash-based adventure games designed to test full story arc completion.<n>We also propose CUA-as-a-Judge, an automated gameplay evaluator, and COAST, an agentic framework leveraging long-term clue memory.<n> Experiments show current GUI agents struggle with full story arcs, while COAST improves milestone completion by bridging the observation-behavior gap.
  arXiv  Detail & Related papers  (2025-09-01T01:33:16Z)
• Screencast-Based Analysis of User-Perceived GUI Responsiveness [53.53923672866705]
  tool is a technique that measures GUI responsiveness directly from mobile screencasts.<n>It uses computer vision to detect user interactions and analyzes frame-level visual changes to compute two key metrics.<n>tool has been deployed in an industrial testing pipeline and analyzes thousands of screencasts daily.
  arXiv  Detail & Related papers  (2025-08-02T12:13:50Z)
• Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation [83.92224427735859]
  We introduce a pre-operative critic mechanism that provides effective feedback prior to the actual execution.<n>We develop a reasoning-bootstrapping based data collection pipeline to create a GUI-Critic-Train and a GUI-Critic-Test.<n>Our model offers significant advantages in critic accuracy compared to current MLLMs.
  arXiv  Detail & Related papers  (2025-06-05T04:12:36Z)
• ViMo: A Generative Visual GUI World Model for App Agent [60.27668506731929]
  ViMo is a visual world model designed to generate future App observations as images. We propose a novel data representation, the Symbolic Text Representation, to overlay text content with symbolic placeholders. With this design, ViMo employs a STR Predictor to predict future GUIs' graphics and a GUI-text Predictor for generating the corresponding text.
  arXiv  Detail & Related papers  (2025-04-15T14:03:10Z)
• GUIWatcher: Automatically Detecting GUI Lags by Analyzing Mobile Application Screencasts [9.997570370503617]
  The Graphical User Interface (GUI) plays a central role in mobile applications, directly affecting usability and user satisfaction. Poor GUI performance, such as lag or unresponsiveness, can lead to negative user experience and decreased mobile application (app) ratings. We present GUIWatcher, a framework designed to detect GUI lags by analyzing screencasts recorded during mobile app testing.
  arXiv  Detail & Related papers  (2025-02-06T16:43:51Z)
• GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration [56.58744345634623]
  We propose GUI-Bee, an MLLM-based autonomous agent, to collect high-quality, environment-specific data through exploration. We also introduce NovelScreenSpot, a benchmark for testing how well the data can help align GUI action grounding models to novel environments.
  arXiv  Detail & Related papers  (2025-01-23T18:16:21Z)
• Zero-Shot Prompting Approaches for LLM-based Graphical User Interface Generation [53.1000575179389]
  We propose a Retrieval-Augmented GUI Generation (RAGG) approach, integrated with an LLM-based GUI retrieval re-ranking and filtering mechanism. In addition, we adapt Prompt Decomposition (PDGG) and Self-Critique (SCGG) for GUI generation. Our evaluation, which encompasses over 3,000 GUI annotations from over 100 crowd-workers with UI/UX experience, shows that SCGG, in contrast to PDGG and RAGG, can lead to more effective GUI generation.
  arXiv  Detail & Related papers  (2024-12-15T22:17:30Z)
• GFT: Graph Foundation Model with Transferable Tree Vocabulary [52.17804507458509]
  We propose a cross-task, cross-domain graph foundation model named GFT, short for Graph Foundation model with transferable Tree vocabulary. By treating computation trees as tokens within the transferable vocabulary, GFT improves model generalization and reduces the risk of negative transfer. The theoretical analyses and extensive experimental studies have demonstrated the transferability of computation trees and shown the effectiveness of GFT across diverse tasks and domains in graph learning.
  arXiv  Detail & Related papers  (2024-11-09T05:14:30Z)
• Vision-driven Automated Mobile GUI Testing via Multimodal Large Language Model [27.97964877860671]
  This paper proposes a vision-driven automated GUI testing approach to detect non-crash functional bugs with Multimodal Large Language Models. It begins by extracting GUI text information and aligning it with screenshots to form a vision prompt, enabling MLLM to understand GUI context. VisionDroid identifies 29 new bugs on Google Play, of which 19 have been confirmed and fixed.
  arXiv  Detail & Related papers  (2024-07-03T11:58:09Z)
• GraphStorm: all-in-one graph machine learning framework for industry applications [75.23076561638348]
  GraphStorm is an end-to-end solution for scalable graph construction, graph model training and inference. Every component in GraphStorm can operate on graphs with billions of nodes and can scale model training and inference to different hardware without changing any code. GraphStorm has been used and deployed for over a dozen billion-scale industry applications after its release in May 2023.
  arXiv  Detail & Related papers  (2024-06-10T04:56:16Z)
• Gamified GUI testing with Selenium in the IntelliJ IDE: A Prototype Plugin [0.559239450391449]
  This paper presents GIPGUT: a prototype of a gamification plugin for IntelliJ IDEA. The plugin enhances testers' engagement with typically monotonous and tedious tasks through achievements, rewards, and profile customization. The results indicate high usability and positive reception of the gamification elements.
  arXiv  Detail & Related papers  (2024-03-14T20:11:11Z)
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [61.93058781222079]
  We develop a flexible question-answering framework targeting real-world textual graphs. We introduce the first retrieval-augmented generation (RAG) approach for general textual graphs. G-Retriever performs RAG over a graph by formulating this task as a Prize-Collecting Steiner Tree optimization problem.
  arXiv  Detail & Related papers  (2024-02-12T13:13:04Z)
• ADA-GAD: Anomaly-Denoised Autoencoders for Graph Anomaly Detection [84.0718034981805]
  We introduce a novel framework called Anomaly-Denoised Autoencoders for Graph Anomaly Detection (ADA-GAD) In the first stage, we design a learning-free anomaly-denoised augmentation method to generate graphs with reduced anomaly levels. In the next stage, the decoders are retrained for detection on the original graph.
  arXiv  Detail & Related papers  (2023-12-22T09:02:01Z)
• On Using GUI Interaction Data to Improve Text Retrieval-based Bug Localization [10.717184444794505]
  We investigate the hypothesis that, for end user-facing applications, connecting information in a bug report with information from the GUI, can improve upon existing techniques for bug localization. We source the current largest dataset of fully-localized and reproducible real bugs for Android apps, with corresponding bug reports.
  arXiv  Detail & Related papers  (2023-10-12T07:14:22Z)
• Weak Supervision for Label Efficient Visual Bug Detection [0.0]
  Traditional testing methods, limited by resources, face difficulties in addressing the plethora of potential bugs. We propose a novel method, utilizing unlabeled gameplay and domain-specific augmentations to generate datasets & self-supervised objectives. Our methodology uses weak-supervision to scale datasets for the crafted objectives and facilitates both autonomous and interactive weak-supervision.
  arXiv  Detail & Related papers  (2023-09-20T06:00:02Z)
• Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project [54.20154707138088]
  We focus on constructing visual code defect prediction models that encompass visual code metrics. We test our models using features extracted from the historical agnostic of a AAA video game project. We find that defect prediction models have better performance overall in terms of the area under the ROC curve.
  arXiv  Detail & Related papers  (2023-09-07T00:18:43Z)
• Identifying Defect-Inducing Changes in Visual Code [54.20154707138088]
  "SZZ Visual Code" (SZZ-VC) is an algorithm that finds changes in visual code based on the differences of graphical elements rather than differences of lines to detect defect-inducing changes. We validated the algorithm for an industry-made AAA video game and 20 music visual programming defects across 12 open source projects.
  arXiv  Detail & Related papers  (2023-09-07T00:12:28Z)
• Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors [3.39487428163997]
  We show that large language models can identify which event is buggy in a sequence of textual descriptions of events from a game. Our results show promising results for employing language models to detect video game bugs.
  arXiv  Detail & Related papers  (2022-10-05T18:44:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.