The widening evaluation gap in medical large language model research 2023 to 2026

• URL: http://arxiv.org/abs/2609.11770v1
• Date: Thu, 10 Sep 2026 16:25:01 GMT
• Title: The widening evaluation gap in medical large language model research 2023 to 2026
• Abstract summary: Large language models are superseded every few quarters; clinical evidence takes years.<n> PubMed returned 11,628 records for January 2023 to June 2026 across fourteen clinical domains, growing 45-fold.<n>Migration to newer systems offset only 56% of the drift.<n>Among studies naming a model still under development no design differed from any other.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are superseded every few quarters; clinical evidence takes years. We asked whether medical research is keeping pace with the systems it evaluates. PubMed returned 11,628 records for January 2023 to June 2026 across fourteen clinical domains, growing 45-fold; 2.5% used a randomised, controlled or prospective design. Evaluation lag, from a study's newest named model release to its own publication, widened from 1.33 to 6.08 quarters. Because discontinued models age mechanically, we benchmarked this against a counterfactual holding model composition fixed: migration to newer systems offset only 56% of the drift (95% CI 50-65). Randomised trials evaluated models a median 4.6 quarters older than other designs (P = 3 x 10^-19), yet among studies naming a model still under development no design differed from any other; 62% of randomised trials evaluated a discontinued family. Rigour and currency are in tension, and that tension reflects model selection rather than research timelines.

Related papers

• PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us [0.7084863262134483]
  We present PhenoBench, an executable benchmark built around the Human Phenotype Project.<n>The benchmark defines 90 clinically grounded tasks across 15 domains and 26 input modalities.<n> Measurements showed question- and representation-dependent predictive value, including positive, near-zero, and negative changes in held-out performance.
  arXiv  Detail & Related papers  (2026-09-05T13:15:40Z)
• Model Retirement Creates Reproducibility Risk in Biomedical AI Publications [0.0]
  We searched PubMed for original research articles that applied a specific large language model to a biomedical task.<n>We identified 8,931 paper-model mentions spanning 5,242 unique publications.<n>Overall, 42% involved a model that was already retired by the time of official publication or is scheduled to retire within two years of publication.
  arXiv  Detail & Related papers  (2026-09-04T04:00:54Z)
• Simulating clinical interventions with a generative multimodal model of human physiology [49.828582089394445]
  HealthFormer is a decoder-only transformer that models the human physiological trajectory generatively.<n>We tokenise each participant's health trajectory across 667 measurements spanning seven domains.<n>HealthFormer improves prediction for 27 of 30 incident-disease and mortality endpoints, exceeding established clinical risk scores in every comparison.
  arXiv  Detail & Related papers  (2026-04-30T14:10:30Z)
• Explainable Admission-Level Predictive Modeling for Prolonged Hospital Stay in Elderly Populations: Challenges in Low- and Middle-Income Countries [65.4286079244589]
  Prolonged length of stay (pLoS) is a significant factor associated with the risk of adverse in-hospital events.<n>We develop and explain a predictive model for pLos using admission-level patient and hospital administrative data.
  arXiv  Detail & Related papers  (2026-01-07T23:35:24Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• Using Pre-training and Interaction Modeling for ancestry-specific disease prediction in UK Biobank [69.90493129893112]
  Recent genome-wide association studies (GWAS) have uncovered the genetic basis of complex traits, but show an under-representation of non-European descent individuals. Here, we assess whether we can improve disease prediction across diverse ancestries using multiomic data.
  arXiv  Detail & Related papers  (2024-04-26T16:39:50Z)
• TotalSegmentator: robust segmentation of 104 anatomical structures in CT images [48.50994220135258]
  We present a deep learning segmentation model for body CT images. The model can segment 104 anatomical structures relevant for use cases such as organ volumetry, disease characterization, and surgical or radiotherapy planning.
  arXiv  Detail & Related papers  (2022-08-11T15:16:40Z)
• Identifying and mitigating bias in algorithms used to manage patients in a pandemic [4.756860520861679]
  Logistic regression models were created to predict COVID-19 mortality, ventilator status and inpatient status using a real-world dataset. Models showed a 57% decrease in the number of biased trials. After calibration, the average sensitivity of the predictive models increased from 0.527 to 0.955.
  arXiv  Detail & Related papers  (2021-10-30T21:10:56Z)
• On the explainability of hospitalization prediction on a large COVID-19 patient dataset [45.82374977939355]
  We develop various AI models to predict hospitalization on a large (over 110$k$) cohort of COVID-19 positive-tested US patients. Despite high data unbalance, the models reach average precision 0.96-0.98 (0.75-0.85), recall 0.96-0.98 (0.74-0.85), and $F_score 0.97-0.98 (0.79-0.83) on the non-hospitalized (or hospitalized) class.
  arXiv  Detail & Related papers  (2021-10-28T10:23:38Z)
• Developing and validating multi-modal models for mortality prediction in COVID-19 patients: a multi-center retrospective study [1.5308395762165423]
  We develop and validate multi-modal models for COVID-19 mortality prediction using multi-center patient data. Our goal is to create a toolkit that would assist investigators and organizations in building multi-modal models for prediction, classification and/or optimization.
  arXiv  Detail & Related papers  (2021-09-01T04:46:27Z)
• Inter-Series Attention Model for COVID-19 Forecasting [23.58411106491221]
  We develop a new neural forecasting model, called Attention Crossing Time Series (textbfACTS), that makes forecasts via comparing patterns across time series obtained from multiple regions. Among 13 out of 18 testings including forecasting newly confirmed cases, hospitalizations and deaths, textbfACTS outperforms all the leading COVID-19 forecasters highlighted by CDC.
  arXiv  Detail & Related papers  (2020-10-25T00:11:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.