Testing Research Software: An In-Depth Survey of Practices, Methods, and Tools

• URL: http://arxiv.org/abs/2501.17739v1
• Date: Wed, 29 Jan 2025 16:27:13 GMT
• Title: Testing Research Software: An In-Depth Survey of Practices, Methods, and Tools
• Authors: Nasir U. Eisty, Upulee Kanewala, Jeffrey C. Carver,
• Abstract summary: Testing research software is challenging due to the software's complexity and to the unique culture of the research software community. This study focuses on test case design, challenges with expected outputs, use of quality metrics, execution methods, tools, and desired tool features.
• Score: 3.831549883667425
• License:
• Abstract: Context: Research software is essential for developing advanced tools and models to solve complex research problems and drive innovation across domains. Therefore, it is essential to ensure its correctness. Software testing plays a vital role in this task. However, testing research software is challenging due to the software's complexity and to the unique culture of the research software community. Aims: Building on previous research, this study provides an in-depth investigation of testing practices in research software, focusing on test case design, challenges with expected outputs, use of quality metrics, execution methods, tools, and desired tool features. Additionally, we explore whether demographic factors influence testing processes. Method: We survey research software developers to understand how they design test cases, handle output challenges, use metrics, execute tests, and select tools. Results: Research software testing varies widely. The primary challenges are test case design, evaluating test quality, and evaluating the correctness of test outputs. Overall, research software developers are not familiar with existing testing tools and have a need for new tools to support their specific needs. Conclusion: Allocating human resources to testing and providing developers with knowledge about effective testing techniques are important steps toward improving the testing process of research software. While many industrial testing tools exist, they are inadequate for testing research software due to its complexity, specialized algorithms, continuous updates, and need for flexible, custom testing approaches. Access to a standard set of testing tools that address these special characteristics will increase level of testing in research software development and reduce the overhead of distributing knowledge about software testing.

Related papers

• The Role of Artificial Intelligence and Machine Learning in Software Testing [0.14896196009851972]
  Artificial Intelligence (AI) and Machine Learning (ML) have significantly impacted various industries. Software testing, a crucial part of the software development lifecycle (SDLC), ensures the quality and reliability of software products. This paper explores the role of AI and ML in software testing by reviewing existing literature, analyzing current tools and techniques, and presenting case studies.
  arXiv  Detail & Related papers  (2024-09-04T13:25:13Z)
• A Comprehensive Study on Automated Testing with the Software Lifecycle [0.6144680854063939]
  The research examines how automated testing makes it easier to evaluate software quality, how it saves time as compared to manual testing, and how it differs from each of them in terms of benefits and drawbacks. The process of testing software applications is simplified, customized to certain testing situations, and can be successfully carried out by using automated testing tools.
  arXiv  Detail & Related papers  (2024-05-02T06:30:37Z)
• Survey of Computerized Adaptive Testing: A Machine Learning Perspective [66.26687542572974]
  Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees. This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method.
  arXiv  Detail & Related papers  (2024-03-31T15:09:47Z)
• Elevating Software Quality in Agile Environments: The Role of Testing Professionals in Unit Testing [0.0]
  Testing is an essential quality activity in the software development process. This paper explores the participation of test engineers in unit testing within an industrial context.
  arXiv  Detail & Related papers  (2024-03-20T00:41:49Z)
• Software Testing and Code Refactoring: A Survey with Practitioners [3.977213079821398]
  This study aims to explore how software testing professionals deal with code to understand the benefits and limitations of this practice in the context of software testing. We concluded that in the context of software testing, offers several benefits, such as supporting the maintenance of automated tests and improving the performance of the testing team. Our study raises discussions on the importance of having testing professionals implement in the code of automated tests, allowing them to improve their coding abilities.
  arXiv  Detail & Related papers  (2023-10-03T01:07:39Z)
• Using Machine Learning To Identify Software Weaknesses From Software Requirement Specifications [49.1574468325115]
  This research focuses on finding an efficient machine learning algorithm to identify software weaknesses from requirement specifications. Keywords extracted using latent semantic analysis help map the CWE categories to PROMISE_exp. Naive Bayes, support vector machine (SVM), decision trees, neural network, and convolutional neural network (CNN) algorithms were tested.
  arXiv  Detail & Related papers  (2023-08-10T13:19:10Z)
• Towards Automatic Generation of Amplified Regression Test Oracles [44.45138073080198]
  We propose a test oracle derivation approach to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT's intended behaviour.
  arXiv  Detail & Related papers  (2023-07-28T12:38:44Z)
• Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning [65.12226891589592]
  This paper proposes a new approach to automated game validation and testing. Our method leverages a data-driven imitation learning technique, which requires little effort and time and no knowledge of machine learning or programming.
  arXiv  Detail & Related papers  (2022-08-15T11:08:44Z)
• Fairness Testing: A Comprehensive Survey and Analysis of Trends [30.637712832450525]
  Unfair behaviors of Machine Learning (ML) software have garnered increasing attention and concern among software engineers. This paper offers a comprehensive survey of existing studies in this field.
  arXiv  Detail & Related papers  (2022-07-20T22:41:38Z)
• SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning [62.997667081978825]
  Testing video games is an increasingly difficult task as traditional methods fail to scale with growing software systems. We present SUPERNOVA, a system responsible for test selection and defect prevention while also functioning as an automation hub. The direct impact of this has been observed to be a reduction in 55% or more testing hours for an undisclosed sports game title.
  arXiv  Detail & Related papers  (2022-03-10T00:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.