Generative AI and Empirical Software Engineering: A Paradigm Shift

• URL: http://arxiv.org/abs/2502.08108v2
• Date: Mon, 03 Nov 2025 09:09:05 GMT
• Title: Generative AI and Empirical Software Engineering: A Paradigm Shift
• Authors: Christoph Treude, Margaret-Anne Storey,
• Abstract summary: This vision paper examines how the integration of large language models into software engineering disrupts established research paradigms.<n>We discuss how it transforms the phenomena we study, the methods and theories we rely on, the data we analyze, and the threats to validity that arise in dynamic AI-mediated environments.<n>Our aim is to help the empirical software engineering community adapt its questions, instruments, and validation standards to a future in which AI systems are not merely tools, but active collaborators shaping software engineering and its study.
• Score: 9.284024538100063
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The adoption of large language models (LLMs) and autonomous agents in software engineering marks an enduring paradigm shift. These systems create new opportunities for tool design, workflow orchestration, and empirical observation, while fundamentally reshaping the roles of developers and the artifacts they produce. Although traditional empirical methods remain central to software engineering research, the rapid evolution of AI introduces new data modalities, alters causal assumptions, and challenges foundational constructs such as "developer", "artifact", and "interaction". As humans and AI agents increasingly co-create, the boundaries between social and technical actors blur, and the reproducibility of findings becomes contingent on model updates and prompt contexts. This vision paper examines how the integration of LLMs into software engineering disrupts established research paradigms. We discuss how it transforms the phenomena we study, the methods and theories we rely on, the data we analyze, and the threats to validity that arise in dynamic AI-mediated environments. Our aim is to help the empirical software engineering community adapt its questions, instruments, and validation standards to a future in which AI systems are not merely tools, but active collaborators shaping software engineering and its study.

Related papers

• Advances and Frontiers of LLM-based Issue Resolution in Software Engineering: A Comprehensive Survey [59.3507264893654]
  Issue resolution is a complex Software Engineering task integral to real-world development.<n> benchmarks like SWE-bench revealed this task as profoundly difficult for large language models.<n>This paper presents a systematic survey of this emerging domain.
  arXiv  Detail & Related papers  (2026-01-15T18:55:03Z)
• Learning to Code with Context: A Study-Based Approach [0.28675177318965045]
  The rapid emergence of generative AI tools is transforming the way software is developed.<n>Project-based courses offer an effective environment to explore and evaluate the integration of AI assistance into real-world development practices.<n>This paper presents our approach and a user study conducted within a university programming project in which students collaboratively developed computer games.
  arXiv  Detail & Related papers  (2025-12-04T20:40:36Z)
• AI- and Ontology-Based Enhancements to FMEA for Advanced Systems Engineering: Current Developments and Future Directions [1.1254693939127907]
  We review recent advances aimed at transforming traditional Mode and Effects Analysis (FMEA) into a more intelligent, data-driven, and semantically enriched process.<n>We examine how techniques from Artificial Intelligence (AI), including machine learning and natural language processing, can transform FMEA into a more dynamic, data-driven process.
  arXiv  Detail & Related papers  (2025-11-21T19:51:06Z)
• Accelerating scientific discovery with the common task framework [48.92654976046941]
  Machine learning (ML) and artificial intelligence (AI) algorithms are transforming the characterization and control of dynamic systems in the engineering, physical, and biological sciences.<n>These emerging modeling paradigms require comparative metrics to evaluate a diverse set of scientific objectives.<n>We introduce a common task framework (CTF) for science and engineering, which features a growing collection of challenge data sets.
  arXiv  Detail & Related papers  (2025-11-06T02:53:07Z)
• A Survey of Vibe Coding with Large Language Models [93.88284590533242]
  "Vibe Coding" is a development methodology where developers validate AI-generated implementations through outcome observation.<n>Despite its transformative potential, the effectiveness of this emergent paradigm remains under-explored.<n>This survey provides the first comprehensive and systematic review of Vibe Coding with large language models.
  arXiv  Detail & Related papers  (2025-10-14T11:26:56Z)
• Generative AI and the Transformation of Software Development Practices [0.0]
  Generative AI is reshaping how software is designed, written, and maintained.<n>This paper examines how AI-assisted techniques are changing software engineering practice.
  arXiv  Detail & Related papers  (2025-10-12T22:02:10Z)
• A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence [87.08051686357206]
  Large Language Models (LLMs) have demonstrated strong capabilities but remain fundamentally static.<n>As LLMs are increasingly deployed in open-ended, interactive environments, this static nature has become a critical bottleneck.<n>This survey provides the first systematic and comprehensive review of self-evolving agents.
  arXiv  Detail & Related papers  (2025-07-28T17:59:05Z)
• Software Reuse in the Generative AI Era: From Cargo Cult Towards AI Native Software Engineering [2.7808182112731537]
  We discuss the implications of AI-assisted generative software reuse in the context of emerging "AI native" software engineering.<n>This paper defines a tentative research agenda and call to action for tackling some of the central issues associated with this approach.
  arXiv  Detail & Related papers  (2025-06-22T08:09:25Z)
• Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI [0.36868085124383626]
  Review presents a comprehensive analysis of two emerging paradigms in AI-assisted software development: vibe coding and agentic coding.<n> Vibe coding emphasizes intuitive, human-in-the-loop interaction through prompt-based, conversational interaction.<n>Agentic coding enables autonomous software development through goal-driven agents capable of planning, executing, testing, and iterating tasks with minimal human intervention.
  arXiv  Detail & Related papers  (2025-05-26T03:00:21Z)
• Challenges and Paths Towards AI for Software Engineering [55.95365538122656]
  We discuss progress in AI for software engineering in threefold manner. First, we provide a structured taxonomy of concrete tasks in AI for software engineering. Second, we outline several key bottlenecks that limit current approaches.
  arXiv  Detail & Related papers  (2025-03-28T17:17:57Z)
• Data Analysis in the Era of Generative AI [56.44807642944589]
  This paper explores the potential of AI-powered tools to reshape data analysis, focusing on design considerations and challenges. We explore how the emergence of large language and multimodal models offers new opportunities to enhance various stages of data analysis workflow. We then examine human-centered design principles that facilitate intuitive interactions, build user trust, and streamline the AI-assisted analysis workflow across multiple apps.
  arXiv  Detail & Related papers  (2024-09-27T06:31:03Z)
• Next-Gen Software Engineering. Big Models for AI-Augmented Model-Driven Software Engineering [0.0]
  The paper provides an overview of the current state of AI-augmented software engineering and develops a corresponding taxonomy, AI4SE.<n>A vision of AI-assisted Big Models in SE is put forth, with the aim of capitalising on the advantages inherent to both approaches in the context of software development.
  arXiv  Detail & Related papers  (2024-09-26T16:49:57Z)
• Bridging Gaps, Building Futures: Advancing Software Developer Diversity and Inclusion Through Future-Oriented Research [50.545824691484796]
  We present insights from SE researchers and practitioners on challenges and solutions regarding diversity and inclusion in SE. We share potential utopian and dystopian visions of the future and provide future research directions and implications for academia and industry.
  arXiv  Detail & Related papers  (2024-04-10T16:18:11Z)
• Charting a Path to Efficient Onboarding: The Role of Software Visualization [49.1574468325115]
  The present study aims to explore the familiarity of managers, leaders, and developers with software visualization tools. This approach incorporated quantitative and qualitative analyses of data collected from practitioners using questionnaires and semi-structured interviews.
  arXiv  Detail & Related papers  (2024-01-17T21:30:45Z)
• Navigating the Complexity of Generative AI Adoption in Software Engineering [6.190511747986327]
  The adoption patterns of Generative Artificial Intelligence (AI) tools within software engineering are investigated. Influencing factors at the individual, technological, and societal levels are analyzed.
  arXiv  Detail & Related papers  (2023-07-12T11:05:19Z)
• Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
  We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering. In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)
• Building an AI-ready RSE Workforce [0.0]
  Machine learning and deep learning are being applied in every aspect of the research software development lifecycles. We discuss our views on today's challenges and opportunities that AI has presented on research software development and engineers.
  arXiv  Detail & Related papers  (2021-11-09T02:36:24Z)
• Human-Robot Collaboration and Machine Learning: A Systematic Review of Recent Research [69.48907856390834]
  Human-robot collaboration (HRC) is the approach that explores the interaction between a human and a robot. This paper proposes a thorough literature review of the use of machine learning techniques in the context of HRC.
  arXiv  Detail & Related papers  (2021-10-14T15:14:33Z)
• Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review [0.0]
  Data scientists often develop machine learning models to solve a variety of problems in the industry and academy. This paper is an effort to investigate the challenges and practices that emerge during the development of ML models from the software engineering perspective.
  arXiv  Detail & Related papers  (2021-02-15T14:25:13Z)
• Technology Readiness Levels for Machine Learning Systems [107.56979560568232]
  Development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. We have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" framework defines a principled process to ensure robust, reliable, and responsible systems.
  arXiv  Detail & Related papers  (2021-01-11T15:54:48Z)
• Software engineering for artificial intelligence and machine learning software: A systematic literature review [6.681725960709127]
  This study aims to investigate how software engineering has been applied in the development of AI/ML systems. Main challenges faced by professionals are in areas of testing, AI software quality, and data management.
  arXiv  Detail & Related papers  (2020-11-07T11:06:28Z)
• Technology Readiness Levels for AI & ML [79.22051549519989]
  Development of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. Engineering systems follow well-defined processes and testing standards to streamline development for high-quality, reliable results. We propose a proven systems engineering approach for machine learning development and deployment.
  arXiv  Detail & Related papers  (2020-06-21T17:14:34Z)
• Machine Learning for Software Engineering: A Systematic Mapping [73.30245214374027]
  The software development industry is rapidly adopting machine learning for transitioning modern day software systems towards highly intelligent and self-learning systems. No comprehensive study exists that explores the current state-of-the-art on the adoption of machine learning across software engineering life cycle stages. This study introduces a machine learning for software engineering (MLSE) taxonomy classifying the state-of-the-art machine learning techniques according to their applicability to various software engineering life cycle stages.
  arXiv  Detail & Related papers  (2020-05-27T11:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.