Evaluation of software impact designed for biomedical research: Are we
measuring what's meaningful?
- URL: http://arxiv.org/abs/2306.03255v1
- Date: Mon, 5 Jun 2023 21:15:05 GMT
- Title: Evaluation of software impact designed for biomedical research: Are we
measuring what's meaningful?
- Authors: Awan Afiaz (1 and 2), Andrey Ivanov (3), John Chamberlin (4), David
Hanauer (5), Candace Savonen (2), Mary J Goldman (6), Martin Morgan (7),
Michael Reich (8), Alexander Getka (9), Aaron Holmes (10 and 11 and 12 and
13), Sarthak Pati (9), Dan Knight (10 and 11 and 12 and 13), Paul C. Boutros
(10 and 11 and 12 and 13), Spyridon Bakas (9), J. Gregory Caporaso (14),
Guilherme Del Fiol (15), Harry Hochheiser (16), Brian Haas (17), Patrick D.
Schloss (18), James A. Eddy (19), Jake Albrecht (19), Andrey Fedorov (20),
Levi Waldron (21), Ava M. Hoffman (2), Richard L. Bradshaw (15), Jeffrey T.
Leek (2) and Carrie Wright (2) ((1) Department of Biostatistics, University
of Washington, Seattle, WA, (2) Biostatistics Program, Public Health Sciences
Division, Fred Hutchinson Cancer Center, Seattle, WA, (3) Department of
Pharmacology and Chemical Biology, Emory University School of Medicine, Emory
University, Atlanta, GA, (4) Department of Biomedical Informatics, University
of Utah, Salt Lake City, UT, (5) Department of Learning Health Sciences,
University of Michigan Medical School, Ann Arbor, MI, (6) University of
California Santa Cruz, Santa Cruz, CA, (7) Roswell Park Comprehensive Cancer
Center, Buffalo, NY, (8) University of California, San Diego, La Jolla, CA,
(9) University of Pennsylvania, Philadelphia, PA, (10) Jonsson Comprehensive
Cancer Center, University of California, Los Angeles, CA, (11) Institute for
Precision Health, University of California, Los Angeles, CA, (12) Department
of Human Genetics, University of California, Los Angeles, CA, (13) Department
of Urology, University of California, Los Angeles, CA, (14) Pathogen and
Microbiome Institute, Northern Arizona University, Flagstaff, AZ, (15)
Department of Biomedical Informatics, University of Utah, Salt Lake City, UT,
(16) Department of Biomedical Informatics, University of Pittsburgh,
Pittsburgh, PA, (17) Methods Development Laboratory, Broad Institute,
Cambridge, MA, (18) Department of Microbiology and Immunology, University of
Michigan, Ann Arbor, MI, (19) Sage Bionetworks, Seattle, WA, (20) Department
of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston,
MA, (21) Department of Epidemiology and Biostatistics, City University of New
York Graduate School of Public Health and Health Policy, New York, NY)
- Abstract summary: Analysis of usage and impact metrics can help developers determine user and community engagement.
There are challenges associated with these analyses including distorted or misleading metrics.
Some tools may be especially beneficial to a small audience, yet may not have compelling typical usage metrics.
- Score: 17.645303073710732
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Software is vital for the advancement of biology and medicine. Analysis of
usage and impact metrics can help developers determine user and community
engagement, justify additional funding, encourage additional use, identify
unanticipated use cases, and help define improvement areas. However, there are
challenges associated with these analyses including distorted or misleading
metrics, as well as ethical and security concerns. More attention to the
nuances involved in capturing impact across the spectrum of biological software
is needed. Furthermore, some tools may be especially beneficial to a small
audience, yet may not have compelling typical usage metrics. We propose more
general guidelines, as well as strategies for more specific types of software.
We highlight outstanding issues regarding how communities measure or evaluate
software impact. To get a deeper understanding of current practices for
software evaluations, we performed a survey of participants in the Informatics
Technology for Cancer Research (ITCR) program funded by the National Cancer
Institute (NCI). We also investigated software among this community and others
to assess how often infrastructure that supports such evaluations is
implemented and how this impacts rates of papers describing usage of the
software. We find that developers recognize the utility of analyzing software
usage, but struggle to find the time or funding for such analyses. We also find
that infrastructure such as social media presence, more in-depth documentation,
the presence of software health metrics, and clear information on how to
contact developers seem to be associated with increased usage rates. Our
findings can help scientific software developers make the most out of
evaluations of their software.
Related papers
- Ethical software requirements from user reviews: A systematic literature review [0.0]
This SLR aims to identify and analyze existing ethical requirements identification and elicitation techniques.
Ethical requirements gathering has recently driven drastic interest in the research community due to the rise of ML and AI-based approaches in decision-making within software applications.
arXiv Detail & Related papers (2024-09-18T19:56:19Z) - How to Measure Performance in Agile Software Development? A Mixed-Method Study [2.477589198476322]
The study aims to identify challenges that arise when using agile software development performance metrics in practice.
Results show that while widely used performance metrics are widely used in practice, agile software development teams face challenges due to a lack of transparency and standardization as well as insufficient accuracy.
arXiv Detail & Related papers (2024-07-08T19:53:01Z) - Efficacy of static analysis tools for software defect detection on open-source projects [0.0]
The study used popular analysis tools such as SonarQube, PMD, Checkstyle, and FindBugs to perform the comparison.
The study results show that SonarQube performs considerably well than all other tools in terms of its defect detection.
arXiv Detail & Related papers (2024-05-20T19:05:32Z) - Charting a Path to Efficient Onboarding: The Role of Software
Visualization [49.1574468325115]
The present study aims to explore the familiarity of managers, leaders, and developers with software visualization tools.
This approach incorporated quantitative and qualitative analyses of data collected from practitioners using questionnaires and semi-structured interviews.
arXiv Detail & Related papers (2024-01-17T21:30:45Z) - Analyzing the Influence of Processor Speed and Clock Speed on Remaining Useful Life Estimation of Software Systems [0.9831489366502301]
This research extends the analysis to assess how changes in environmental attributes, such as operating system and clock speed, affect RUL estimation in software.
Findings are rigorously validated using real performance data from controlled test beds and compared with predictive model-generated data.
This exploration yields actionable knowledge for software maintenance and optimization strategies.
arXiv Detail & Related papers (2023-09-22T04:46:34Z) - Using Machine Learning To Identify Software Weaknesses From Software
Requirement Specifications [49.1574468325115]
This research focuses on finding an efficient machine learning algorithm to identify software weaknesses from requirement specifications.
Keywords extracted using latent semantic analysis help map the CWE categories to PROMISE_exp. Naive Bayes, support vector machine (SVM), decision trees, neural network, and convolutional neural network (CNN) algorithms were tested.
arXiv Detail & Related papers (2023-08-10T13:19:10Z) - LLM-based Interaction for Content Generation: A Case Study on the
Perception of Employees in an IT department [85.1523466539595]
This paper presents a questionnaire survey to identify the intention to use generative tools by employees of an IT company.
Our results indicate a rather average acceptability of generative tools, although the more useful the tool is perceived to be, the higher the intention seems to be.
Our analyses suggest that the frequency of use of generative tools is likely to be a key factor in understanding how employees perceive these tools in the context of their work.
arXiv Detail & Related papers (2023-04-18T15:35:43Z) - Understanding metric-related pitfalls in image analysis validation [59.15220116166561]
This work provides the first comprehensive common point of access to information on pitfalls related to validation metrics in image analysis.
Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy.
arXiv Detail & Related papers (2023-02-03T14:57:40Z) - Towards a Fair Comparison and Realistic Design and Evaluation Framework
of Android Malware Detectors [63.75363908696257]
We analyze 10 influential research works on Android malware detection using a common evaluation framework.
We identify five factors that, if not taken into account when creating datasets and designing detectors, significantly affect the trained ML models.
We conclude that the studied ML-based detectors have been evaluated optimistically, which justifies the good published results.
arXiv Detail & Related papers (2022-05-25T08:28:08Z) - AI Explainability 360: Impact and Design [120.95633114160688]
In 2019, we created AI Explainability 360 (Arya et al. 2020), an open source software toolkit featuring ten diverse and state-of-the-art explainability methods.
This paper examines the impact of the toolkit with several case studies, statistics, and community feedback.
The paper also describes the flexible design of the toolkit, examples of its use, and the significant educational material and documentation available to its users.
arXiv Detail & Related papers (2021-09-24T19:17:09Z) - Investigating Software Usage in the Social Sciences: A Knowledge Graph
Approach [0.483420384410068]
We present SoftwareKG, a knowledge graph that contains information about software mentions from more than 51,000 scientific articles from the social sciences.
A neural network was used to train an LSTM based neural network to identify software mentions in scientific articles.
We show how SoftwareKG can be used to assess the role of software in the social sciences.
arXiv Detail & Related papers (2020-03-24T08:38:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.