Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChat

• URL: http://arxiv.org/abs/2409.07829v1
• Date: Thu, 12 Sep 2024 08:25:33 GMT
• Title: Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChat
• Authors: Sidong Feng, Haochuan Lu, Jianqin Jiang, Ting Xiong, Likun Huang, Yinglin Liang, Xiaoqin Li, Yuetang Deng, Aldeida Aleti,
• Abstract summary: We introduce CAT to create cost-effective UI automation tests for industry apps by combining machine learning and Large Language Models. CAT then employs machine learning techniques, with LLMs serving as a complementary, to map the target element on the UI screen. Our evaluations on the WeChat testing dataset demonstrate the CAT's performance and cost-effectiveness, achieving 90% UI automation with $0.34 cost.
• Score: 8.80569452545511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: UI automation tests play a crucial role in ensuring the quality of mobile applications. Despite the growing popularity of machine learning techniques to generate these tests, they still face several challenges, such as the mismatch of UI elements. The recent advances in Large Language Models (LLMs) have addressed these issues by leveraging their semantic understanding capabilities. However, a significant gap remains in applying these models to industrial-level app testing, particularly in terms of cost optimization and knowledge limitation. To address this, we introduce CAT to create cost-effective UI automation tests for industry apps by combining machine learning and LLMs with best practices. Given the task description, CAT employs Retrieval Augmented Generation (RAG) to source examples of industrial app usage as the few-shot learning context, assisting LLMs in generating the specific sequence of actions. CAT then employs machine learning techniques, with LLMs serving as a complementary optimizer, to map the target element on the UI screen. Our evaluations on the WeChat testing dataset demonstrate the CAT's performance and cost-effectiveness, achieving 90% UI automation with $0.34 cost, outperforming the state-of-the-art. We have also integrated our approach into the real-world WeChat testing platform, demonstrating its usefulness in detecting 141 bugs and enhancing the developers' testing process.

Related papers

• AutoPT: How Far Are We from the End2End Automated Web Penetration Testing? [54.65079443902714]
  We introduce AutoPT, an automated penetration testing agent based on the principle of PSM driven by LLMs. Our results show that AutoPT outperforms the baseline framework ReAct on the GPT-4o mini model.
  arXiv  Detail & Related papers  (2024-11-02T13:24:30Z)
• Skill-Adpative Imitation Learning for UI Test Reuse [13.538724823517292]
  We propose a skill-adaptive imitation learning framework designed to enhance the effectiveness of UI test migration. Results show that SAIL substantially improves the effectiveness of UI test migration, with 149% higher success rate than state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-09-20T08:13:04Z)
• Automated Text Scoring in the Age of Generative AI for the GPU-poor [49.1574468325115]
  We analyze the performance and efficiency of open-source, small-scale generative language models for automated text scoring. Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance.
  arXiv  Detail & Related papers  (2024-07-02T01:17:01Z)
• Test Oracle Automation in the era of LLMs [52.69509240442899]
  Large Language Models (LLMs) have demonstrated remarkable proficiency in tackling diverse software testing tasks. This paper aims to enable discussions on the potential of using LLMs for test oracle automation, along with the challenges that may emerge during the generation of various types of oracles.
  arXiv  Detail & Related papers  (2024-05-21T13:19:10Z)
• Automating REST API Postman Test Cases Using LLM [0.0]
  This research paper is dedicated to the exploration and implementation of an automated approach to generate test cases using Large Language Models. The methodology integrates the use of Open AI to enhance the efficiency and effectiveness of test case generation. The model that is developed during the research is trained using manually collected postman test cases or instances for various Rest APIs.
  arXiv  Detail & Related papers  (2024-04-16T15:53:41Z)
• ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation [30.693616802332745]
  This paper presents a novel benchmark, AssistGUI, to evaluate whether models are capable of manipulating the mouse and keyboard on the Windows platform in response to user-requested tasks. We propose an advanced Actor-Critic framework, which incorporates a sophisticated GUI driven by an AI agent and adept at handling lengthy procedural tasks.
  arXiv  Detail & Related papers  (2023-12-20T15:28:38Z)
• LLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities [8.504639288314063]
  Test script generation is a vital component of software testing, enabling efficient and reliable automation of repetitive test tasks. Existing generation approaches often encounter limitations, such as difficulties in accurately capturing and reproducing test scripts across diverse devices, platforms, and applications. This paper investigates the application of large language models (LLM) in the domain of mobile application test script generation.
  arXiv  Detail & Related papers  (2023-09-24T07:58:57Z)
• OverPrompt: Enhancing ChatGPT through Efficient In-Context Learning [49.38867353135258]
  We propose OverPrompt, leveraging the in-context learning capability of LLMs to handle multiple task inputs. Our experiments show that OverPrompt can achieve cost-efficient zero-shot classification without causing significant detriment to task performance.
  arXiv  Detail & Related papers  (2023-05-24T10:08:04Z)
• Benchmarking Automated Machine Learning Methods for Price Forecasting Applications [58.720142291102135]
  We show the possibility of substituting manually created ML pipelines with automated machine learning (AutoML) solutions. Based on the CRISP-DM process, we split the manual ML pipeline into a machine learning and non-machine learning part. We show in a case study for the industrial use case of price forecasting, that domain knowledge combined with AutoML can weaken the dependence on ML experts.
  arXiv  Detail & Related papers  (2023-04-28T10:27:38Z)
• ALBench: A Framework for Evaluating Active Learning in Object Detection [102.81795062493536]
  This paper contributes an active learning benchmark framework named as ALBench for evaluating active learning in object detection. Developed on an automatic deep model training system, this ALBench framework is easy-to-use, compatible with different active learning algorithms, and ensures the same training and testing protocols.
  arXiv  Detail & Related papers  (2022-07-27T07:46:23Z)
• The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study [15.016047591601094]
  We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges. ML generates input for system, GUI, unit, performance, and testing or improves the performance of existing generation methods.
  arXiv  Detail & Related papers  (2022-06-21T09:26:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.