Model Retirement Creates Reproducibility Risk in Biomedical AI Publications

• URL: http://arxiv.org/abs/2609.04699v1
• Date: Fri, 04 Sep 2026 04:00:54 GMT
• Title: Model Retirement Creates Reproducibility Risk in Biomedical AI Publications
• Abstract summary: We searched PubMed for original research articles that applied a specific large language model to a biomedical task.<n>We identified 8,931 paper-model mentions spanning 5,242 unique publications.<n>Overall, 42% involved a model that was already retired by the time of official publication or is scheduled to retire within two years of publication.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background. Large language models (LLMs) are being adopted in biomedical research at a rapid and accelerating pace, yet commercial services that host many widely used models operate under deprecation schedules that can complicate scientific reproducibility. Methods. We searched PubMed for original research articles from 2022 through March 2026 that applied a specific LLM to a biomedical task. An extraction agent identified model names from 61,077 article abstracts with human reviewers validating a subset for extraction accuracy. Extracted model names were normalized to canonical model identifiers. Lifecycle data (release date, retirement date, status) were compiled for the 50 most frequently used models. Results. We identified 8,931 paper-model mentions spanning 5,242 unique publications after restricting the analysis to the 50 most frequently used models. Among these mentions, 77.7% cited a commercial closed-weight model. Overall, 42% involved a model that was already retired by the time of official publication or is scheduled to retire within two years of publication. The median interval from publication to model retirement was 538 days. Conclusion. Many biomedical publications using LLMs are on a trajectory toward computational non-reproducibility after publication. Model deprecation should be treated as a core reporting and preservation issue for biomedical research.

Related papers

• The widening evaluation gap in medical large language model research 2023 to 2026 [0.0]
  Large language models are superseded every few quarters; clinical evidence takes years.<n> PubMed returned 11,628 records for January 2023 to June 2026 across fourteen clinical domains, growing 45-fold.<n>Migration to newer systems offset only 56% of the drift.<n>Among studies naming a model still under development no design differed from any other.
  arXiv  Detail & Related papers  (2026-09-10T16:25:01Z)
• AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era [55.37558486468993]
  We introduce AgentIdeaBench, a benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration.<n>We report matched Static-Active evaluations for 33 LLMs across 40 densely scored subfields spanning five disciplines.<n>Active exploration reveals considerably more capability headroom, and that headroom is unevenly distributed across models.
  arXiv  Detail & Related papers  (2026-09-07T15:19:13Z)
• The Half-Lives of Generative-AI Evidence: A 40-Record Audit, a Claim-Currency Framework, and a Reflexive Case of Frontier-Model-Assisted Research [0.0]
  Generative-AI evaluations can become historical before publication, yet calendar age does not affect every conclusion equally.<n>This paper audits a maximum-variation purposive corpus of 40 empirical records appearing between 18 July 2025 and 17 July 2026.<n>The audit coded publication route, execution timing, model identity, age of the newest named generation or immutable snapshot, same-family supersession and refresh behaviour.
  arXiv  Detail & Related papers  (2026-07-27T05:59:49Z)
• Large Language Model-Based Agents for Automated Research Reproducibility: An Exploratory Study in Alzheimer's Disease [1.9938547353667109]
  We used the "Quick Access" dataset of the National Alzheimer's Coordinating Center.<n>We identified highly cited published research manuscripts using NACC data.<n>We created a simulated research team of LLM-based autonomous agents tasked with writing and executing code.
  arXiv  Detail & Related papers  (2025-05-29T01:31:55Z)
• Biomedical Foundation Model: A Survey [84.26268124754792]
  Foundation models are large-scale pre-trained models that learn from extensive unlabeled datasets.<n>These models can be adapted to various applications such as question answering and visual understanding.<n>This survey explores the potential of foundation models across diverse domains within biomedical fields.
  arXiv  Detail & Related papers  (2025-03-03T22:42:00Z)
• Gender-Neutral Large Language Models for Medical Applications: Reducing Bias in PubMed Abstracts [3.7906296809297406]
  This paper presents a pipeline for mitigating gender bias in large language models used in medical literature by neutralizing gendered occupational pronouns.<n>A dataset of 379,000 PubMed abstracts from 1965 was processed to identify and modify pronouns tied to professions.
  arXiv  Detail & Related papers  (2025-01-10T22:07:56Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text [82.7001841679981]
  BioMedLM is a 2.7 billion parameter GPT-style autoregressive model trained exclusively on PubMed abstracts and full articles. When fine-tuned, BioMedLM can produce strong multiple-choice biomedical question-answering results competitive with larger models. BioMedLM can also be fine-tuned to produce useful answers to patient questions on medical topics.
  arXiv  Detail & Related papers  (2024-03-27T10:18:21Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• Boosting Adverse Drug Event Normalization on Social Media: General-Purpose Model Initialization and Biomedical Semantic Text Similarity Benefit Zero-Shot Linking in Informal Contexts [0.32228025627337864]
  We propose a new approach for adverse drug event normalization on social media texts. Our experimental results on several social media datasets demonstrate the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2023-07-31T21:12:06Z)
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks [68.39821375903591]
  Generalist AI holds the potential to address limitations due to its versatility in interpreting different data types. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model.
  arXiv  Detail & Related papers  (2023-05-26T17:14:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.