Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research

• URL: http://arxiv.org/abs/2506.12691v1
• Date: Sun, 15 Jun 2025 02:25:50 GMT
• Title: Get on the Train or be Left on the Station: Using LLMs for Software Engineering Research
• Authors: Bianca Trinkenreich, Fabio Calefato, Geir Hanssen, Kelly Blincoe, Marcos Kalinowski, Mauro Pezzè, Paolo Tell, Margaret-Anne Storey,
• Abstract summary: We argue that the SE research community must proactively engage with and shape the integration of Large Language Models into research practices.<n>This position paper employs McLuhan's Tetrad of Media Laws to analyze the impact of LLMs on SE research.
• Score: 11.38769375570817
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The adoption of Large Language Models (LLMs) is not only transforming software engineering (SE) practice but is also poised to fundamentally disrupt how research is conducted in the field. While perspectives on this transformation range from viewing LLMs as mere productivity tools to considering them revolutionary forces, we argue that the SE research community must proactively engage with and shape the integration of LLMs into research practices, emphasizing human agency in this transformation. As LLMs rapidly become integral to SE research - both as tools that support investigations and as subjects of study - a human-centric perspective is essential. Ensuring human oversight and interpretability is necessary for upholding scientific rigor, fostering ethical responsibility, and driving advancements in the field. Drawing from discussions at the 2nd Copenhagen Symposium on Human-Centered AI in SE, this position paper employs McLuhan's Tetrad of Media Laws to analyze the impact of LLMs on SE research. Through this theoretical lens, we examine how LLMs enhance research capabilities through accelerated ideation and automated processes, make some traditional research practices obsolete, retrieve valuable aspects of historical research approaches, and risk reversal effects when taken to extremes. Our analysis reveals opportunities for innovation and potential pitfalls that require careful consideration. We conclude with a call to action for the SE research community to proactively harness the benefits of LLMs while developing frameworks and guidelines to mitigate their risks, to ensure continued rigor and impact of research in an AI-augmented future.

Related papers

• Recognising, Anticipating, and Mitigating LLM Pollution of Online Behavioural Research [20.76449931945918]
  We identify three interacting variants through which LLM Pollution threatens online behavioural research.<n>We propose a multi-layered response spanning researcher practices, platform accountability, and community efforts.
  arXiv  Detail & Related papers  (2025-08-02T14:40:54Z)
• Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities [62.05713042908654]
  This paper provides a review of advances in Large Language Models (LLMs) alignment through the lens of inverse reinforcement learning (IRL)<n>We highlight the necessity of constructing neural reward models from human data and discuss the formal and practical implications of this paradigm shift.
  arXiv  Detail & Related papers  (2025-07-17T14:22:24Z)
• Roadmap for using large language models (LLMs) to accelerate cross-disciplinary research with an example from computational biology [0.0]
  Large language models (LLMs) are powerful artificial intelligence (AI) tools transforming how research is conducted.<n>Their use in research has been met with skepticism, due to concerns about hallucinations, biases and potential harms to research.<n>Here, we present a roadmap for integrating LLMs into cross-disciplinary research.
  arXiv  Detail & Related papers  (2025-07-04T17:20:14Z)
• Scaling and Beyond: Advancing Spatial Reasoning in MLLMs Requires New Recipes [84.1059652774853]
  Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in general vision-language tasks.<n>Recent studies have exposed critical limitations in their spatial reasoning capabilities.<n>This deficiency in spatial reasoning significantly constrains MLLMs' ability to interact effectively with the physical world.
  arXiv  Detail & Related papers  (2025-04-21T11:48:39Z)
• Towards Evaluation Guidelines for Empirical Studies involving LLMs [6.174354685766166]
  Large language models (LLMs) have changed the software engineering research landscape.<n>This paper contributes the first set of holistic guidelines for such studies.
  arXiv  Detail & Related papers  (2024-11-12T09:35:23Z)
• A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law [65.87885628115946]
  Large language models (LLMs) are revolutionizing the landscapes of finance, healthcare, and law. We highlight the instrumental role of LLMs in enhancing diagnostic and treatment methodologies in healthcare, innovating financial analytics, and refining legal interpretation and compliance strategies. We critically examine the ethics for LLM applications in these fields, pointing out the existing ethical concerns and the need for transparent, fair, and robust AI systems.
  arXiv  Detail & Related papers  (2024-05-02T22:43:02Z)
• ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models [56.08917291606421]
  ResearchAgent is an AI-based system for ideation and operationalization of novel work.<n>ResearchAgent automatically defines novel problems, proposes methods and designs experiments, while iteratively refining them.<n>We experimentally validate our ResearchAgent on scientific publications across multiple disciplines.
  arXiv  Detail & Related papers  (2024-04-11T13:36:29Z)
• Large Language Model Alignment: A Survey [42.03229317132863]
  The potential of large language models (LLMs) is undeniably vast; however, they may yield texts that are imprecise, misleading, or even detrimental. This survey endeavors to furnish an extensive exploration of alignment methodologies designed for LLMs. We also probe into salient issues including the models' interpretability, and potential vulnerabilities to adversarial attacks.
  arXiv  Detail & Related papers  (2023-09-26T15:49:23Z)
• A Comprehensive Overview of Large Language Models [68.22178313875618]
  Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks. This article provides an overview of the existing literature on a broad range of LLM-related concepts.
  arXiv  Detail & Related papers  (2023-07-12T20:01:52Z)
• Learnings from Frontier Development Lab and SpaceML -- AI Accelerators for NASA and ESA [57.06643156253045]
  Research with AI and ML technologies lives in a variety of settings with often asynchronous goals and timelines. We perform a case study of the Frontier Development Lab (FDL), an AI accelerator under a public-private partnership from NASA and ESA. FDL research follows principled practices that are grounded in responsible development, conduct, and dissemination of AI research.
  arXiv  Detail & Related papers  (2020-11-09T21:23:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.