Large-scale information retrieval in software engineering -- an
experience report from industrial application
- URL: http://arxiv.org/abs/2308.11750v1
- Date: Tue, 22 Aug 2023 19:30:56 GMT
- Title: Large-scale information retrieval in software engineering -- an
experience report from industrial application
- Authors: Michael Unterkalmsteiner, Tony Gorschek, Robert Feldt, Niklas Lavesson
- Abstract summary: We describe an engineering task, test case selection, and illustrate our problem analysis and solution discovery process.
We analyze, in the context of the studied company, how test case selection is performed and design a series of experiments evaluating the performance of different IR techniques.
- Score: 9.62054859086279
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Software Engineering activities are information intensive. Research proposes
Information Retrieval (IR) techniques to support engineers in their daily
tasks, such as establishing and maintaining traceability links, fault
identification, and software maintenance. We describe an engineering task, test
case selection, and illustrate our problem analysis and solution discovery
process. The objective of the study is to gain an understanding of to what
extent IR techniques (one potential solution) can be applied to test case
selection and provide decision support in a large-scale, industrial setting. We
analyze, in the context of the studied company, how test case selection is
performed and design a series of experiments evaluating the performance of
different IR techniques. Each experiment provides lessons learned from
implementation, execution, and results, feeding to its successor. The three
experiments led to the following observations: 1) there is a lack of research
on scalable parameter optimization of IR techniques for software engineering
problems; 2) scaling IR techniques to industry data is challenging, in
particular for latent semantic analysis; 3) the IR context poses constraints on
the empirical evaluation of IR techniques, requiring more research on
developing valid statistical approaches. We believe that our experiences in
conducting a series of IR experiments with industry grade data are valuable for
peer researchers so that they can avoid the pitfalls that we have encountered.
Furthermore, we identified challenges that need to be addressed in order to
bridge the gap between laboratory IR experiments and real applications of IR in
the industry.
Related papers
- Teaching Design Science as a Method for Effective Research Development [0.24578723416255752]
Applying Design Science Research (DSR) methodology is becoming a popular working resource for Information Systems (IS) and Software engineering studies.
This chapter includes examples of DSR, a teaching methodology, learning objectives, and recommendations.
We have created a survey artifact intended to gather data on the experiences of design science users.
arXiv Detail & Related papers (2024-07-13T10:43:06Z) - Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective [111.58315434849047]
robustness of neural information retrieval models (IR) models has garnered significant attention.
We view the robustness of IR to be a multifaceted concept, emphasizing its necessity against adversarial attacks, out-of-distribution (OOD) scenarios and performance variance.
We provide an in-depth discussion of existing methods, datasets, and evaluation metrics, shedding light on challenges and future directions in the era of large language models.
arXiv Detail & Related papers (2024-07-09T16:07:01Z) - DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents [49.74065769505137]
We introduce DISCOVERYWORLD, the first virtual environment for developing and benchmarking an agent's ability to perform complete cycles of novel scientific discovery.
It includes 120 different challenge tasks spanning eight topics each with three levels of difficulty and several parametric variations.
We find that strong baseline agents, that perform well in prior published environments, struggle on most DISCOVERYWORLD tasks.
arXiv Detail & Related papers (2024-06-10T20:08:44Z) - The Causal Chambers: Real Physical Systems as a Testbed for AI Methodology [10.81691411087626]
In some fields of AI, machine learning and statistics, the validation of new methods and algorithms is often hindered by the scarcity of suitable real-world datasets.
We have constructed two devices that allow us to quickly and inexpensively produce large datasets from non-trivial but well-understood physical systems.
arXiv Detail & Related papers (2024-04-17T13:00:52Z) - A Systematic Literature Review on Explainability for Machine/Deep
Learning-based Software Engineering Research [23.966640472958105]
This paper presents a systematic literature review of approaches that aim to improve the explainability of AI models within the context of Software Engineering.
We aim to summarize the SE tasks where XAI techniques have shown success to date; (2) classify and analyze different XAI techniques; and (3) investigate existing evaluation approaches.
arXiv Detail & Related papers (2024-01-26T03:20:40Z) - Information Retrieval Meets Large Language Models: A Strategic Report
from Chinese IR Community [180.28262433004113]
Large Language Models (LLMs) have demonstrated exceptional capabilities in text understanding, generation, and knowledge inference.
LLMs and humans form a new technical paradigm that is more powerful for information seeking.
To thoroughly discuss the transformative impact of LLMs on IR research, the Chinese IR community conducted a strategic workshop in April 2023.
arXiv Detail & Related papers (2023-07-19T05:23:43Z) - Pitfalls in Experiments with DNN4SE: An Analysis of the State of the
Practice [0.7614628596146599]
We conduct a mapping study, examining 194 experiments with techniques that rely on deep neural networks appearing in 55 papers published in premier software engineering venues.
Our study reveals that most of the experiments, including those that have received ACM artifact badges, have fundamental limitations that raise doubts about the reliability of their findings.
arXiv Detail & Related papers (2023-05-19T09:55:48Z) - Wizard of Search Engine: Access to Information Through Conversations
with Search Engines [58.53420685514819]
We make efforts to facilitate research on CIS from three aspects.
We formulate a pipeline for CIS with six sub-tasks: intent detection (ID), keyphrase extraction (KE), action prediction (AP), query selection (QS), passage selection (PS) and response generation (RG)
We release a benchmark dataset, called wizard of search engine (WISE), which allows for comprehensive and in-depth research on all aspects of CIS.
arXiv Detail & Related papers (2021-05-18T06:35:36Z) - Artificial Intelligence for IT Operations (AIOPS) Workshop White Paper [50.25428141435537]
Artificial Intelligence for IT Operations (AIOps) is an emerging interdisciplinary field arising in the intersection between machine learning, big data, streaming analytics, and the management of IT operations.
Main aim of the AIOPS workshop is to bring together researchers from both academia and industry to present their experiences, results, and work in progress in this field.
arXiv Detail & Related papers (2021-01-15T10:43:10Z) - Neural Methods for Effective, Efficient, and Exposure-Aware Information
Retrieval [7.3371176873092585]
We present novel neural architectures and methods motivated by the specific needs and challenges of information retrieval.
In many real-life IR tasks, the retrieval involves extremely large collections--such as the document index of a commercial Web search engine--containing billions of documents.
arXiv Detail & Related papers (2020-12-21T21:20:16Z) - AutoOD: Automated Outlier Detection via Curiosity-guided Search and
Self-imitation Learning [72.99415402575886]
Outlier detection is an important data mining task with numerous practical applications.
We propose AutoOD, an automated outlier detection framework, which aims to search for an optimal neural network model.
Experimental results on various real-world benchmark datasets demonstrate that the deep model identified by AutoOD achieves the best performance.
arXiv Detail & Related papers (2020-06-19T18:57:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.