Automated User Experience Testing through Multi-Dimensional Performance Impact Analysis

• URL: http://arxiv.org/abs/2104.03453v1
• Date: Thu, 8 Apr 2021 01:18:01 GMT
• Title: Automated User Experience Testing through Multi-Dimensional Performance Impact Analysis
• Authors: Chidera Biringa, Gokhan Kul
• Abstract summary: We propose a novel automated user experience testing methodology. It learns how code changes impact the time unit and system tests take, and extrapolates user experience changes based on this information. Our open-source tool achieved 3.7% mean absolute error rate with a random forest regressor.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although there are many automated software testing suites, they usually focus on unit, system, and interface testing. However, especially software updates such as new security features have the potential to diminish user experience. In this paper, we propose a novel automated user experience testing methodology that learns how code changes impact the time unit and system tests take, and extrapolate user experience changes based on this information. Such a tool can be integrated into existing continuous integration pipelines, and it provides software teams immediate user experience feedback. We construct a feature set from lexical, layout, and syntactic characteristics of the code, and using Abstract Syntax Tree-Based Embeddings, we can calculate the approximate semantic distance to feed into a machine learning algorithm. In our experiments, we use several regression methods to estimate the time impact of software updates. Our open-source tool achieved 3.7% mean absolute error rate with a random forest regressor.

Related papers

• Towards Realistic Evaluation of Commit Message Generation by Matching Online and Offline Settings [77.20838441870151]
  Commit message generation is a crucial task in software engineering that is challenging to evaluate correctly. We use an online metric - the number of edits users introduce before committing the generated messages to the VCS - to select metrics for offline experiments. Our results indicate that edit distance exhibits the highest correlation, whereas commonly used similarity metrics such as BLEU and METEOR demonstrate low correlation.
  arXiv  Detail & Related papers  (2024-10-15T20:32:07Z)
• Which Combination of Test Metrics Can Predict Success of a Software Project? A Case Study in a Year-Long Project Course [1.553083901660282]
  Testing plays an important role in securing the success of a software development project. We investigate whether we can quantify the effects various types of testing have on functional suitability.
  arXiv  Detail & Related papers  (2024-08-22T04:23:51Z)
• A Comprehensive Study on Automated Testing with the Software Lifecycle [0.6144680854063939]
  The research examines how automated testing makes it easier to evaluate software quality, how it saves time as compared to manual testing, and how it differs from each of them in terms of benefits and drawbacks. The process of testing software applications is simplified, customized to certain testing situations, and can be successfully carried out by using automated testing tools.
  arXiv  Detail & Related papers  (2024-05-02T06:30:37Z)
• Artificial intelligence for context-aware visual change detection in software test automation [5.174422378856116]
  We introduce a novel graph-based method for visual change detection in software test automation. Our method accurately identifies UI controls from software screenshots and constructs a graph representing contextual and spatial relationships between the controls. It can accurately detect visual software changes in various simple and complex test scenarios.
  arXiv  Detail & Related papers  (2024-05-01T21:22:33Z)
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback [78.60644407028022]
  We introduce MINT, a benchmark that evaluates large language models' ability to solve tasks with multi-turn interactions. LLMs generally benefit from tools and language feedback, with performance gains of 1-8% for each turn of tool use. LLMs evaluated, supervised instruction-finetuning (SIFT) and reinforcement learning from human feedback (RLHF) generally hurt multi-turn capabilities.
  arXiv  Detail & Related papers  (2023-09-19T15:25:42Z)
• TestLab: An Intelligent Automated Software Testing Framework [0.0]
  TestLab is an automated software testing framework that attempts to gather a set of testing methods and automate them using Artificial Intelligence. The first two modules aim to identify vulnerabilities from different perspectives, while the third module enhances traditional automated software testing by automatically generating test cases.
  arXiv  Detail & Related papers  (2023-06-06T11:45:22Z)
• ConvLab-3: A Flexible Dialogue System Toolkit Based on a Unified Data Format [88.33443450434521]
  Task-oriented dialogue (TOD) systems function as digital assistants, guiding users through various tasks such as booking flights or finding restaurants. Existing toolkits for building TOD systems often fall short of in delivering comprehensive arrays of data, models, and experimental environments. We introduce ConvLab-3: a multifaceted dialogue system toolkit crafted to bridge this gap.
  arXiv  Detail & Related papers  (2022-11-30T16:37:42Z)
• ALBench: A Framework for Evaluating Active Learning in Object Detection [102.81795062493536]
  This paper contributes an active learning benchmark framework named as ALBench for evaluating active learning in object detection. Developed on an automatic deep model training system, this ALBench framework is easy-to-use, compatible with different active learning algorithms, and ensures the same training and testing protocols.
  arXiv  Detail & Related papers  (2022-07-27T07:46:23Z)
• SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning [62.997667081978825]
  Testing video games is an increasingly difficult task as traditional methods fail to scale with growing software systems. We present SUPERNOVA, a system responsible for test selection and defect prevention while also functioning as an automation hub. The direct impact of this has been observed to be a reduction in 55% or more testing hours for an undisclosed sports game title.
  arXiv  Detail & Related papers  (2022-03-10T00:47:46Z)
• Agents for Automated User Experience Testing [4.6453787256723365]
  We propose an agent based approach for automatic UX testing. We develop agents with basic problem solving skills and a core affect model. Although this research is still at a primordial state, we believe the results here make a strong case for the use of intelligent agents.
  arXiv  Detail & Related papers  (2021-04-13T14:13:28Z)
• Integrated Benchmarking and Design for Reproducible and Accessible Evaluation of Robotic Agents [61.36681529571202]
  We describe a new concept for reproducible robotics research that integrates development and benchmarking. One of the central components of this setup is the Duckietown Autolab, a standardized setup that is itself relatively low-cost and reproducible. We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.
  arXiv  Detail & Related papers  (2020-09-09T15:31:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.