Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models

• URL: http://arxiv.org/abs/2410.09012v1
• Date: Fri, 11 Oct 2024 17:27:04 GMT
• Title: Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models
• Authors: Hao Li, Cor-Paul Bezemer, Ahmed E. Hassan,
• Abstract summary: We analyze 155 FM4SE and 997 SE4FM blog posts from leading technology companies. We observed that while code generation is the most prominent FM4SE task, FMs are leveraged for many other SE activities. Although the emphasis is on cloud deployments, there is a growing interest in compressing FMs and deploying them on smaller devices.
• Score: 11.993910471523073
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models (FMs) such as large language models (LLMs) have significantly impacted many fields, including software engineering (SE). The interaction between SE and FMs has led to the integration of FMs into SE practices (FM4SE) and the application of SE methodologies to FMs (SE4FM). While several literature surveys exist on academic contributions to these trends, we are the first to provide a practitioner's view. We analyze 155 FM4SE and 997 SE4FM blog posts from leading technology companies, leveraging an FM-powered surveying approach to systematically label and summarize the discussed activities and tasks. We observed that while code generation is the most prominent FM4SE task, FMs are leveraged for many other SE activities such as code understanding, summarization, and API recommendation. The majority of blog posts on SE4FM are about model deployment & operation, and system architecture & orchestration. Although the emphasis is on cloud deployments, there is a growing interest in compressing FMs and deploying them on smaller devices such as edge or mobile devices. We outline eight future research directions inspired by our gained insights, aiming to bridge the gap between academic findings and real-world applications. Our study not only enriches the body of knowledge on practical applications of FM4SE and SE4FM but also demonstrates the utility of FMs as a powerful and efficient approach in conducting literature surveys within technical and grey literature domains. Our dataset, results, code and used prompts can be found in our online replication package at https://github.com/SAILResearch/fmse-blogs.

Related papers

• On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards [11.99718417371013]
  This research focuses on understanding how these FM leaderboards operate in real-world scenarios ("leaderboard operations") We identify 5 unique workflow patterns and develop a domain model that outlines the essential components and their interaction within FM leaderboards. We then identify 8 unique types of leaderboard smells in LBOps.
  arXiv  Detail & Related papers  (2024-07-04T17:12:00Z)
• Synergizing Foundation Models and Federated Learning: A Survey [23.416321895575507]
  This paper discusses the potentials and challenges of synergizing Federated Learning (FL) and Foundation Models (FM) FL is a collaborative learning paradigm that breaks the barrier of data availability from different participants. It provides a promising solution to customize and adapt FMs to a wide range of domain-specific tasks using distributed datasets whilst preserving privacy.
  arXiv  Detail & Related papers  (2024-06-18T17:58:09Z)
• Rethinking Software Engineering in the Foundation Model Era: A Curated Catalogue of Challenges in the Development of Trustworthy FMware [13.21876203209586]
  We identify 10 key SE4FMware challenges that have caused enterprise FMware development to be unproductive, costly, and risky. We present FMArts, which is our long-term effort towards creating a cradle-to-grave platform for the engineering of trustworthy FMware.
  arXiv  Detail & Related papers  (2024-02-25T00:53:16Z)
• Progress and Opportunities of Foundation Models in Bioinformatics [77.74411726471439]
  Foundations models (FMs) have ushered in a new era in computational biology, especially in the realm of deep learning. Central to our focus is the application of FMs to specific biological problems, aiming to guide the research community in choosing appropriate FMs for their research needs. Review analyses challenges and limitations faced by FMs in biology, such as data noise, model explainability, and potential biases.
  arXiv  Detail & Related papers  (2024-02-06T02:29:17Z)
• Learn From Model Beyond Fine-Tuning: A Survey [78.80920533793595]
  Learn From Model (LFM) focuses on the research, modification, and design of foundation models (FM) based on the model interface. The study of LFM techniques can be broadly categorized into five major areas: model tuning, model distillation, model reuse, meta learning and model editing. This paper gives a comprehensive review of the current methods based on FM from the perspective of LFM.
  arXiv  Detail & Related papers  (2023-10-12T10:20:36Z)
• The Role of Federated Learning in a Wireless World with Foundation Models [59.8129893837421]
  Foundation models (FMs) are general-purpose artificial intelligence (AI) models that have recently enabled multiple brand-new generative AI applications. Currently, the exploration of the interplay between FMs and federated learning (FL) is still in its nascent stage. This article explores the extent to which FMs are suitable for FL over wireless networks, including a broad overview of research challenges and opportunities.
  arXiv  Detail & Related papers  (2023-10-06T04:13:10Z)
• VideoGLUE: Video General Understanding Evaluation of Foundation Models [89.07145427268948]
  We evaluate video understanding capabilities of foundation models (FMs) using a carefully designed experiment protocol. We jointly profile FMs' hallmark and efficacy efficiency when adapting to general video understanding tasks.
  arXiv  Detail & Related papers  (2023-07-06T17:47:52Z)
• Multimodal Recommender Systems: A Survey [50.23505070348051]
  Multimodal Recommender System (MRS) has attracted much attention from both academia and industry recently. In this paper, we will give a comprehensive survey of the MRS models, mainly from technical views. To access more details of the surveyed papers, such as implementation code, we open source a repository.
  arXiv  Detail & Related papers  (2023-02-08T05:12:54Z)
• FETA: Towards Specializing Foundation Models for Expert Task Applications [49.57393504125937]
  Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization. We show in this paper that FMs still have poor out-of-the-box performance on expert tasks. We propose a first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation.
  arXiv  Detail & Related papers  (2022-09-08T08:47:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.