LLMs for Science: Usage for Code Generation and Data Analysis

• URL: http://arxiv.org/abs/2311.16733v4
• Date: Tue, 23 Apr 2024 08:12:46 GMT
• Title: LLMs for Science: Usage for Code Generation and Data Analysis
• Authors: Mohamed Nejjar, Luca Zacharias, Fabian Stiehle, Ingo Weber,
• Abstract summary: Large language models (LLMs) have been touted to enable increased productivity in many areas of today's work life. It is still unclear how the potential of LLMs will materialise in research practice.
• Score: 0.07499722271664144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have been touted to enable increased productivity in many areas of today's work life. Scientific research as an area of work is no exception: the potential of LLM-based tools to assist in the daily work of scientists has become a highly discussed topic across disciplines. However, we are only at the very onset of this subject of study. It is still unclear how the potential of LLMs will materialise in research practice. With this study, we give first empirical evidence on the use of LLMs in the research process. We have investigated a set of use cases for LLM-based tools in scientific research, and conducted a first study to assess to which degree current tools are helpful. In this paper we report specifically on use cases related to software engineering, such as generating application code and developing scripts for data analytics. While we studied seemingly simple use cases, results across tools differ significantly. Our results highlight the promise of LLM-based tools in general, yet we also observe various issues, particularly regarding the integrity of the output these tools provide.

Related papers

• Towards Evaluation Guidelines for Empirical Studies involving LLMs [6.174354685766166]
  Large language models (LLMs) have changed the software engineering research landscape. This paper contributes the first set of guidelines for such studies. Our goal is to start a discussion in the software engineering research community to reach a common understanding of what our community standards are for high-quality empirical studies involving LLMs.
  arXiv  Detail & Related papers  (2024-11-12T09:35:23Z)
• What is the Role of Large Language Models in the Evolution of Astronomy Research? [0.0]
  ChatGPT and other state-of-the-art large language models (LLMs) are rapidly transforming multiple fields. These models, commonly trained on vast datasets, exhibit human-like text generation capabilities.
  arXiv  Detail & Related papers  (2024-09-30T12:42:25Z)
• What Are Tools Anyway? A Survey from the Language Model Perspective [67.18843218893416]
  Language models (LMs) are powerful yet mostly for text generation tasks. We provide a unified definition of tools as external programs used by LMs. We empirically study the efficiency of various tooling methods.
  arXiv  Detail & Related papers  (2024-03-18T17:20:07Z)
• LLMs in the Imaginarium: Tool Learning through Simulated Trial and Error [54.954211216847135]
  Existing large language models (LLMs) only reach a correctness rate in the range of 30% to 60%. We propose a biologically inspired method for tool-augmented LLMs, simulated trial and error (STE) STE orchestrates three key mechanisms for successful tool use behaviors in the biological system: trial and error, imagination, and memory.
  arXiv  Detail & Related papers  (2024-03-07T18:50:51Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows [72.40917624485822]
  We introduce DataDreamer, an open source Python library that allows researchers to implement powerful large language models. DataDreamer also helps researchers adhere to best practices that we propose to encourage open science.
  arXiv  Detail & Related papers  (2024-02-16T00:10:26Z)
• From Prompt Engineering to Prompt Science With Human in the Loop [12.230632679443364]
  This article presents a new methodology inspired by codebook construction through qualitative methods to address that. We show how a set of researchers can work through a rigorous process of labeling, deliberating, and documenting to remove subjectivity and bring transparency and replicability to prompt generation process.
  arXiv  Detail & Related papers  (2024-01-01T01:37:36Z)
• LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis [18.775126929754833]
  Thematic analysis (TA) has been widely used for analyzing qualitative data in many disciplines and fields. Human coders develop and deepen their data interpretation and coding over multiple iterations, making TA labor-intensive and time-consuming. We propose a human-LLM collaboration framework (i.e., LLM-in-the-loop) to conduct TA with in-context learning (ICL)
  arXiv  Detail & Related papers  (2023-10-23T17:05:59Z)
• MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use [82.24774504584066]
  Large language models (LLMs) have garnered significant attention due to their impressive natural language processing (NLP) capabilities. We introduce MetaTool, a benchmark designed to evaluate whether LLMs have tool usage awareness and can correctly choose tools. We conduct experiments involving eight popular LLMs and find that the majority of them still struggle to effectively select tools.
  arXiv  Detail & Related papers  (2023-10-04T19:39:26Z)
• Calculating Originality of LLM Assisted Source Code [0.0]
  We propose a neural network-based tool to determine the original effort (and LLM's contribution) put by students in writing source codes. Our tool is motivated by minimum description length measures like Kolmogorov complexity.
  arXiv  Detail & Related papers  (2023-07-10T11:30:46Z)
• LLM-based Interaction for Content Generation: A Case Study on the Perception of Employees in an IT department [85.1523466539595]
  This paper presents a questionnaire survey to identify the intention to use generative tools by employees of an IT company. Our results indicate a rather average acceptability of generative tools, although the more useful the tool is perceived to be, the higher the intention seems to be. Our analyses suggest that the frequency of use of generative tools is likely to be a key factor in understanding how employees perceive these tools in the context of their work.
  arXiv  Detail & Related papers  (2023-04-18T15:35:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.