On (Mis)perceptions of testing effectiveness: an empirical study
- URL: http://arxiv.org/abs/2402.07222v1
- Date: Sun, 11 Feb 2024 14:50:01 GMT
- Title: On (Mis)perceptions of testing effectiveness: an empirical study
- Authors: Sira Vegas, Patricia Riofrio, Esperanza Marcos, Natalia Juristo
- Abstract summary: This research aims to discover how well the perceptions of the defect detection effectiveness of different techniques match their real effectiveness in the absence of prior experience.
In the original study, we conduct a controlled experiment with students applying two testing techniques and a code review technique.
At the end of the experiment, they take a survey to find out which technique they perceive to be most effective.
The results of the replicated study confirm the findings of the original study and suggest that participants' perceptions might be based not on their opinions about complexity or preferences for techniques but on how well they think that they have applied the techniques.
- Score: 1.8026347864255505
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: A recurring problem in software development is incorrect decision making on
the techniques, methods and tools to be used. Mostly, these decisions are based
on developers' perceptions about them. A factor influencing people's
perceptions is past experience, but it is not the only one. In this research,
we aim to discover how well the perceptions of the defect detection
effectiveness of different techniques match their real effectiveness in the
absence of prior experience. To do this, we conduct an empirical study plus a
replication. During the original study, we conduct a controlled experiment with
students applying two testing techniques and a code review technique. At the
end of the experiment, they take a survey to find out which technique they
perceive to be most effective. The results show that participants' perceptions
are wrong and that this mismatch is costly in terms of quality. In order to
gain further insight into the results, we replicate the controlled experiment
and extend the survey to include questions about participants' opinions on the
techniques and programs. The results of the replicated study confirm the
findings of the original study and suggest that participants' perceptions might
be based not on their opinions about complexity or preferences for techniques
but on how well they think that they have applied the techniques.
Related papers
- Hazards in Deep Learning Testing: Prevalence, Impact and Recommendations [17.824339932321788]
We identify 10 commonly adopted empirical evaluation hazards that may significantly impact experimental results.
Our findings indicate that all 10 hazards have the potential to invalidate experimental findings.
We propose a point set of 10 good empirical practices that has the potential to mitigate the impact of the hazards.
arXiv Detail & Related papers (2023-09-11T11:05:34Z) - A Double Machine Learning Approach to Combining Experimental and Observational Data [59.29868677652324]
We propose a double machine learning approach to combine experimental and observational studies.
Our framework tests for violations of external validity and ignorability under milder assumptions.
arXiv Detail & Related papers (2023-07-04T02:53:11Z) - PyExperimenter: Easily distribute experiments and track results [63.871474825689134]
PyExperimenter is a tool to facilitate the setup, documentation, execution, and subsequent evaluation of results from an empirical study of algorithms.
It is intended to be used by researchers in the field of artificial intelligence, but is not limited to those.
arXiv Detail & Related papers (2023-01-16T10:43:02Z) - Fair Effect Attribution in Parallel Online Experiments [57.13281584606437]
A/B tests serve the purpose of reliably identifying the effect of changes introduced in online services.
It is common for online platforms to run a large number of simultaneous experiments by splitting incoming user traffic randomly.
Despite a perfect randomization between different groups, simultaneous experiments can interact with each other and create a negative impact on average population outcomes.
arXiv Detail & Related papers (2022-10-15T17:15:51Z) - Homophily and Incentive Effects in Use of Algorithms [17.55279695774825]
We present a crowdsourcing vignette study designed to assess the impacts of two plausible factors on AI-informed decision-making.
First, we examine homophily -- do people defer more to models that tend to agree with them?
Second, we consider incentives -- how do people incorporate a (known) cost structure in the hybrid decision-making setting?
arXiv Detail & Related papers (2022-05-19T17:11:04Z) - Sources of Irreproducibility in Machine Learning: A Review [3.905855359082687]
There exist no theoretical framework that relates experiment design choices to potential effects on the conclusions.
The objective of this paper is to develop a framework that enables applied data science practitioners and researchers to understand which experiment design choices can lead to false findings.
arXiv Detail & Related papers (2022-04-15T18:26:03Z) - StudyMe: A New Mobile App for User-Centric N-of-1 Trials [68.8204255655161]
N-of-1 trials are multi-crossover self-experiments that allow individuals to systematically evaluate the effect of interventions on their personal health goals.
We present StudyMe, an open-source mobile application that is freely available from https://play.google.com/store/apps/details?id=health.studyu.me.
arXiv Detail & Related papers (2021-07-31T20:43:36Z) - Black Magic in Deep Learning: How Human Skill Impacts Network Training [24.802914836352738]
We present an initial study based on 31 participants with different levels of experience.
The results show a strong positive correlation between the participant's experience and the final performance.
arXiv Detail & Related papers (2020-08-13T15:56:14Z) - A Survey on Causal Inference [64.45536158710014]
Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics.
Various causal effect estimation methods for observational data have sprung up.
arXiv Detail & Related papers (2020-02-05T21:35:29Z) - Generalization Bounds and Representation Learning for Estimation of
Potential Outcomes and Causal Effects [61.03579766573421]
We study estimation of individual-level causal effects, such as a single patient's response to alternative medication.
We devise representation learning algorithms that minimize our bound, by regularizing the representation's induced treatment group distance.
We extend these algorithms to simultaneously learn a weighted representation to further reduce treatment group distances.
arXiv Detail & Related papers (2020-01-21T10:16:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.