From Release to Adoption: Challenges in Reusing Pre-trained AI Models for Downstream Developers

• URL: http://arxiv.org/abs/2506.23234v2
• Date: Wed, 16 Jul 2025 08:04:59 GMT
• Title: From Release to Adoption: Challenges in Reusing Pre-trained AI Models for Downstream Developers
• Authors: Peerachai Banyongrakkul, Mansooreh Zahedi, Patanamon Thongtanunam, Christoph Treude, Haoyu Gao,
• Abstract summary: Pre-trained models (PTMs) have gained widespread popularity and achieved remarkable success across various fields.<n>However, the challenges faced by downstream developers in reusing PTMs in software systems are less explored.<n>We qualitatively created and analyzed a dataset of 840 PTM-related issue reports from 31 OSS GitHub projects.
• Score: 10.230447995338055
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained models (PTMs) have gained widespread popularity and achieved remarkable success across various fields, driven by their groundbreaking performance and easy accessibility through hosting providers. However, the challenges faced by downstream developers in reusing PTMs in software systems are less explored. To bridge this knowledge gap, we qualitatively created and analyzed a dataset of 840 PTM-related issue reports from 31 OSS GitHub projects. We systematically developed a comprehensive taxonomy of PTM-related challenges that developers face in downstream projects. Our study identifies seven key categories of challenges that downstream developers face in reusing PTMs, such as model usage, model performance, and output quality. We also compared our findings with existing taxonomies. Additionally, we conducted a resolution time analysis and, based on statistical tests, found that PTM-related issues take significantly longer to be resolved than issues unrelated to PTMs, with significant variation across challenge categories. We discuss the implications of our findings for practitioners and possibilities for future research.

Related papers

• Empowering Time Series Analysis with Synthetic Data: A Survey and Outlook in the Era of Foundation Models [104.17057231661371]
  Time series analysis is crucial for understanding dynamics of complex systems.<n>Recent advances in foundation models have led to task-agnostic Time Series Foundation Models (TSFMs) and Large Language Model-based Time Series Models (TSLLMs)<n>Their success depends on large, diverse, and high-quality datasets, which are challenging to build due to regulatory, diversity, quality, and quantity constraints.<n>This survey provides a comprehensive review of synthetic data for TSFMs and TSLLMs, analyzing data generation strategies, their role in model pretraining, fine-tuning, and evaluation, and identifying future research directions.
  arXiv  Detail & Related papers  (2025-03-14T13:53:46Z)
• Towards a Classification of Open-Source ML Models and Datasets for Software Engineering [52.257764273141184]
  Open-source Pre-Trained Models (PTMs) and datasets provide extensive resources for various Machine Learning (ML) tasks. These resources lack a classification tailored to Software Engineering (SE) needs. We apply an SE-oriented classification to PTMs and datasets on a popular open-source ML repository, Hugging Face (HF), and analyze the evolution of PTMs over time.
  arXiv  Detail & Related papers  (2024-11-14T18:52:05Z)
• Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvement [62.94719119451089]
  Lingma SWE-GPT series learns from and simulating real-world code submission activities. Lingma SWE-GPT 72B resolves 30.20% of GitHub issues, marking a significant improvement in automatic issue resolution.
  arXiv  Detail & Related papers  (2024-11-01T14:27:16Z)
• MetaTrading: An Immersion-Aware Model Trading Framework for Vehicular Metaverse Services [94.61039892220037]
  We propose an immersion-aware model trading framework that facilitates data provision for services while ensuring privacy through federated learning (FL)<n>We design an incentive mechanism to incentivize metaverse users (MUs) to contribute high-value models under resource constraints.<n>We develop a fully distributed dynamic reward algorithm based on deep reinforcement learning, without accessing any private information about MUs and other MSPs.
  arXiv  Detail & Related papers  (2024-10-25T16:20:46Z)
• Challenges of Using Pre-trained Models: the Practitioners' Perspective [16.042355796766124]
  We analyze the popularity and difficulty trends of PTM-related questions on Stack Overflow. We find that PTM-related questions are becoming more and more popular over time. This observation emphasizes the significant difficulty and complexity associated with the practical application of PTMs.
  arXiv  Detail & Related papers  (2024-04-23T03:39:14Z)
• PeaTMOSS: A Dataset and Initial Analysis of Pre-Trained Models in Open-Source Software [6.243303627949341]
  This paper presents the PeaTMOSS dataset, which comprises metadata for 281,638 PTMs and detailed snapshots for all PTMs. The dataset includes 44,337 mappings from 15,129 downstream GitHub repositories to the 2,530 PTMs they use. Our analysis provides the first summary statistics for the PTM supply chain, showing the trend of PTM development and common shortcomings of PTM package documentation.
  arXiv  Detail & Related papers  (2024-02-01T15:55:50Z)
• Deep Learning Model Reuse in the HuggingFace Community: Challenges, Benefit and Trends [12.645960268553686]
  The ubiquity of large-scale Pre-Trained Models (PTMs) is on the rise, sparking interest in model hubs and dedicated platforms for hosting PTMs. We present a taxonomy of the challenges and benefits associated with PTM reuse within this community. Our findings highlight prevalent challenges such as limited guidance for beginner users, struggles with model output comprehensibility in training or inference, and a lack of model understanding.
  arXiv  Detail & Related papers  (2024-01-24T01:50:29Z)
• Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis [70.78170766633039]
  We address the need for means of assessing MTS forecasting proposals reliably and fairly. BasicTS+ is a benchmark designed to enable fair, comprehensive, and reproducible comparison of MTS forecasting solutions. We apply BasicTS+ along with rich datasets to assess the capabilities of more than 45 MTS forecasting solutions.
  arXiv  Detail & Related papers  (2023-10-09T19:52:22Z)
• Does Synthetic Data Generation of LLMs Help Clinical Text Mining? [51.205078179427645]
  We investigate the potential of OpenAI's ChatGPT to aid in clinical text mining. We propose a new training paradigm that involves generating a vast quantity of high-quality synthetic data. Our method has resulted in significant improvements in the performance of downstream tasks.
  arXiv  Detail & Related papers  (2023-03-08T03:56:31Z)
• An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry [2.1346819928536687]
  Machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks.
  arXiv  Detail & Related papers  (2023-03-05T02:28:15Z)
• Pre-Trained Models: Past, Present and Future [126.21572378910746]
  Large-scale pre-trained models (PTMs) have recently achieved great success and become a milestone in the field of artificial intelligence (AI) By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch.
  arXiv  Detail & Related papers  (2021-06-14T02:40:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.