Large Language Models Lack Temporal Awareness of Medical Knowledge

• URL: http://arxiv.org/abs/2605.13045v1
• Date: Wed, 13 May 2026 06:04:40 GMT
• Title: Large Language Models Lack Temporal Awareness of Medical Knowledge
• Abstract summary: Existing methods for evaluating medical knowledge of Large Language Models (LLMs) are largely based on atemporal examination-style benchmarks.<n>We build TempoMed-Bench, the first-of-its-kind benchmark for evaluating the temporal awareness of the LLMs in the medical domain through evolving guideline knowledge.
• Score: 30.240452466538073
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The existing methods for evaluating the medical knowledge of Large Language Models (LLMs) are largely based on atemporal examination-style benchmarks, while in reality, medical knowledge is inherently dynamic and continuously evolves as new evidence emerges and treatments are approved. Consequently, evaluating medical knowledge without a temporal context may provide an incomplete assessment of whether LLMs can accurately reason about time-specific medical knowledge. Moreover, most medical data are historical, requiring the models not only to recall the correct knowledge, but also to know when that knowledge is correct. To bridge the gap, we built TempoMed-Bench, the first-of-its-kind benchmark for evaluating the temporal awareness of the LLMs in the medical domain through evolving guideline knowledge. Based on the TempoMed-Bench, our evaluation analysis first reveals that LLMs lack temporal awareness in medical knowledge through the key findings: (1) model performance on up-to-date medical knowledge exhibits a gradual linear decline over time rather than a sharp knowledge-cutoff behavior, suggesting that parametric medical knowledge is not strictly bounded by knowledge cutoffs; (2) LLMs consistently struggle more with recalling outdated historical medical knowledge than with up-to-date recommendations: accuracy of historical knowledge is only 25.37%-53.89% of up-to-date knowledge, indicating potential knowledge forgetting effects during training; and (3) LLMs often exhibit temporally inconsistent behaviors, where predictions fluctuate irregularly across neighboring years. We also show that the temporal awareness problem is a challenge that cannot be easily solved when integrated with agentic search tools (-3.15%-14.14%). This work highlights an important yet underexplored challenge and motivates future research on developing LLMs that can better encode time-specific medical knowledge.

Related papers

• MedCoG: Maximizing LLM Inference Density in Medical Reasoning via Meta-Cognitive Regulation [50.42633168081233]
  Large Language Models (LLMs) have shown strong potential in complex medical reasoning yet face diminishing gains under inference scaling laws.<n>We propose MedCoG, a Medical Meta-Cognition Agent with Knowledge Graph, where the meta-cognitive assessments of task complexity, familiarity, and knowledge density dynamically regulate utilization of procedural, episodic, and factual knowledge.<n>Experiments demonstrate the effectiveness and efficiency of MedCoG on five hard sets of medical benchmarks, yielding 5.5x inference density.
  arXiv  Detail & Related papers  (2026-02-08T10:54:04Z)
• Facts Fade Fast: Evaluating Memorization of Outdated Medical Knowledge in Large Language Models [23.266037521209796]
  Large Language Models show potential to enhance healthcare by assisting medical researchers and physicians.<n>Their reliance on static training data is a major risk when medical recommendations evolve with new research and developments.
  arXiv  Detail & Related papers  (2025-09-04T15:17:50Z)
• Beyond Memorization: A Rigorous Evaluation Framework for Medical Knowledge Editing [72.8373875453882]
  knowledge editing (KE) has emerged as a promising approach to update specific facts in Large Language Models (LLMs) without the need for full retraining.<n>We propose a novel framework called MedEditBench to rigorously evaluate the effectiveness of existing KE methods in the medical domain.<n>Our findings indicate that current KE methods result in only superficial memorization of the injected information, failing to generalize to new scenarios.
  arXiv  Detail & Related papers  (2025-06-04T02:14:43Z)
• Can We Edit LLMs for Long-Tail Biomedical Knowledge? [13.462834780111768]
  We conduct the first comprehensive study to investigate the effectiveness of knowledge editing methods for editing long-tail biomedical knowledge.<n>Our results indicate that, while existing editing methods can enhance LLMs' performance on long-tail biomedical knowledge, their performance on long-tail knowledge remains inferior to that on high-frequency popular knowledge.
  arXiv  Detail & Related papers  (2025-04-14T17:08:20Z)
• Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments [108.55277188617035]
  Large language models (LLMs) have been widely adopted in various downstream task domains, but their abilities to directly recall and apply factual medical knowledge remains under-explored.<n>We introduce the Medical Knowledge Judgment dataset (MKJ), a dataset derived from the Unified Medical Language System (UMLS), a comprehensive repository of standardized vocabularies and knowledge graphs.<n>Through a binary classification framework, MKJ evaluates LLMs' grasp of fundamental medical facts by having them assess the validity of concise, one-hop statements.
  arXiv  Detail & Related papers  (2025-02-20T05:27:51Z)
• Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval [61.70489848327436]
  KARE is a novel framework that integrates knowledge graph (KG) community-level retrieval with large language models (LLMs) reasoning.<n>Extensive experiments demonstrate that KARE outperforms leading models by up to 10.8-15.0% on MIMIC-III and 12.6-12.7% on MIMIC-IV for mortality and readmission predictions.
  arXiv  Detail & Related papers  (2024-10-06T18:46:28Z)
• Reliable and diverse evaluation of LLM medical knowledge mastery [6.825565574784612]
  We propose a novel framework that generates reliable and diverse test samples to evaluate medical-specific LLMs. We use our proposed framework to systematically investigate the mastery of medical factual knowledge of 12 well-known LLMs.
  arXiv  Detail & Related papers  (2024-09-22T03:13:38Z)
• Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models [89.13883089162951]
  Model editing aims to precisely alter the behaviors of large language models (LLMs) in relation to specific knowledge. This approach has proven effective in addressing issues of hallucination and outdated information in LLMs. However, the potential of using model editing to modify knowledge in the medical field remains largely unexplored.
  arXiv  Detail & Related papers  (2024-02-28T06:40:57Z)
• KnowTuning: Knowledge-aware Fine-tuning for Large Language Models [83.5849717262019]
  We propose a knowledge-aware fine-tuning (KnowTuning) method to improve fine-grained and coarse-grained knowledge awareness of LLMs. KnowTuning generates more facts with less factual error rate under fine-grained facts evaluation.
  arXiv  Detail & Related papers  (2024-02-17T02:54:32Z)
• Don't Ignore Dual Logic Ability of LLMs while Privatizing: A Data-Intensive Analysis in Medical Domain [19.46334739319516]
  We study how the dual logic ability of LLMs is affected during the privatization process in the medical domain. Our results indicate that incorporating general domain dual logic data into LLMs not only enhances LLMs' dual logic ability but also improves their accuracy.
  arXiv  Detail & Related papers  (2023-09-08T08:20:46Z)
• Knowledge-tuning Large Language Models with Structured Medical Knowledge Bases for Reliable Response Generation in Chinese [29.389119917322102]
  Large Language Models (LLMs) have demonstrated remarkable success in diverse natural language processing (NLP) tasks in general domains. We propose knowledge-tuning, which leverages structured medical knowledge bases for the LLMs to grasp domain knowledge efficiently. We also release cMedKnowQA, a Chinese medical knowledge question-answering dataset constructed from medical knowledge bases.
  arXiv  Detail & Related papers  (2023-09-08T07:42:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.