How do Pre-Trained Models Support Software Engineering? An Empirical Study in Hugging Face

• URL: http://arxiv.org/abs/2506.03013v1
• Date: Tue, 03 Jun 2025 15:51:17 GMT
• Title: How do Pre-Trained Models Support Software Engineering? An Empirical Study in Hugging Face
• Authors: Alexandra González, Xavier Franch, David Lo, Silverio Martínez-Fernández,
• Abstract summary: Open-Source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks.<n>These resources lack a classification tailored to Software Engineering (SE) needs.<n>We derive a taxonomy encompassing 147 SE tasks and apply an SE-oriented classification to PTMs in a popular open-source ML repository, Hugging Face (HF)<n>We find that code generation is the most common SE task among PTMs, while requirements engineering and software design activities receive limited attention.
• Score: 52.257764273141184
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-Source Pre-Trained Models (PTMs) provide extensive resources for various Machine Learning (ML) tasks, yet these resources lack a classification tailored to Software Engineering (SE) needs. To address this gap, we derive a taxonomy encompassing 147 SE tasks and apply an SE-oriented classification to PTMs in a popular open-source ML repository, Hugging Face (HF). Our repository mining study began with a systematically gathered database of PTMs from the HF API, considering their model card descriptions and metadata, and the abstract of the associated arXiv papers. We confirmed SE relevance through multiple filtering steps: detecting outliers, identifying near-identical PTMs, and the use of Gemini 2.0 Flash, which was validated with five pilot studies involving three human annotators. This approach uncovered 2,205 SE PTMs. We find that code generation is the most common SE task among PTMs, primarily focusing on software implementation, while requirements engineering and software design activities receive limited attention. In terms of ML tasks, text generation dominates within SE PTMs. Notably, the number of SE PTMs has increased markedly since 2023 Q2. Our classification provides a solid foundation for future automated SE scenarios, such as the sampling and selection of suitable PTMs.

Related papers

• Towards a Classification of Open-Source ML Models and Datasets for Software Engineering [52.257764273141184]
  Open-source Pre-Trained Models (PTMs) and datasets provide extensive resources for various Machine Learning (ML) tasks. These resources lack a classification tailored to Software Engineering (SE) needs. We apply an SE-oriented classification to PTMs and datasets on a popular open-source ML repository, Hugging Face (HF), and analyze the evolution of PTMs over time.
  arXiv  Detail & Related papers  (2024-11-14T18:52:05Z)
• Automated categorization of pre-trained models for software engineering: A case study with a Hugging Face dataset [9.218130273952383]
  Software engineering activities have been revolutionized by the advent of pre-trained models (PTMs) The Hugging Face (HF) platform simplifies the use of PTMs by collecting, storing, and curating several models. This paper introduces an approach to enable the automatic classification of PTMs for SE tasks.
  arXiv  Detail & Related papers  (2024-05-21T20:26:17Z)
• PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software [6.243303627949341]
  This paper presents the PeaTMOSS dataset, which comprises metadata for 281,638 PTMs and detailed snapshots for all PTMs. The dataset includes 44,337 mappings from 15,129 downstream GitHub repositories to the 2,530 PTMs they use. Our analysis provides the first summary statistics for the PTM supply chain, showing the trend of PTM development and common shortcomings of PTM package documentation.
  arXiv  Detail & Related papers  (2024-02-01T15:55:50Z)
• TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data [73.29220562541204]
  We consider harnessing the amazing power of language models (LLMs) to solve our task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets.
  arXiv  Detail & Related papers  (2024-01-24T04:28:50Z)
• PeaTMOSS: Mining Pre-Trained Models in Open-Source Software [6.243303627949341]
  We present the PeaTMOSS dataset: Pre-Trained Models in Open-Source Software. PeaTMOSS has three parts: a snapshot of 281,638 PTMs, (2) 27,270 open-source software repositories that use PTMs, and (3) a mapping between PTMs and the projects that use them.
  arXiv  Detail & Related papers  (2023-10-05T15:58:45Z)
• Naming Practices of Pre-Trained Models in Hugging Face [4.956536094440504]
  Pre-Trained Models (PTMs) are used in computer systems to adapt for quality or performance prior to deployment. Researchers publish PTMs, which engineers adapt for quality or performance prior to deployment. Prior research has reported that model names are not always well chosen - and are sometimes erroneous. In this paper, we frame and conduct the first empirical investigation of PTM naming practices in the Hugging Face PTM registry.
  arXiv  Detail & Related papers  (2023-10-02T21:13:32Z)
• Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey [66.18478838828231]
  Multi-modal pre-trained big models have drawn more and more attention in recent years. This paper introduces the background of multi-modal pre-training by reviewing the conventional deep, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network, and knowledge enhanced pre-training.
  arXiv  Detail & Related papers  (2023-02-20T15:34:03Z)
• Ranking and Tuning Pre-trained Models: A New Paradigm of Exploiting Model Hubs [136.4492678691406]
  We propose a new paradigm of exploiting model hubs by ranking and tuning pre-trained models. The best ranked PTM can be fine-tuned and deployed if we have no preference for the model's architecture. The tuning part introduces a novel method for multiple PTMs tuning, which surpasses dedicated methods.
  arXiv  Detail & Related papers  (2021-10-20T12:59:23Z)
• Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews [2.66512000865131]
  We study the accuracy and time efficiency of pre-trained neural language models (PTMs) for app review classification. We set up different studies to evaluate PTMs in multiple settings. In all cases, Micro and Macro Precision, Recall, and F1-scores will be used.
  arXiv  Detail & Related papers  (2021-04-12T23:23:45Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.