MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication

• URL: http://arxiv.org/abs/2601.09853v1
• Date: Wed, 14 Jan 2026 20:23:02 GMT
• Title: MedRedFlag: Investigating how LLMs Redirect Misconceptions in Real-World Health Communication
• Authors: Sraavya Sambara, Yuan Pu, Ayman Ali, Vishala Mishra, Lionel Wong, Monica Agrawal,
• Abstract summary: Real-world health questions from patients often unintentionally embed false assumptions or premises.<n>In such cases, safe medical communication typically involves redirection: addressing the implicit misconception and then responding to the underlying patient context.<n>We investigate how large language models (LLMs) react to false premises embedded within real-world health questions.
• Score: 4.557109813131144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world health questions from patients often unintentionally embed false assumptions or premises. In such cases, safe medical communication typically involves redirection: addressing the implicit misconception and then responding to the underlying patient context, rather than the original question. While large language models (LLMs) are increasingly being used by lay users for medical advice, they have not yet been tested for this crucial competency. Therefore, in this work, we investigate how LLMs react to false premises embedded within real-world health questions. We develop a semi-automated pipeline to curate MedRedFlag, a dataset of 1100+ questions sourced from Reddit that require redirection. We then systematically compare responses from state-of-the-art LLMs to those from clinicians. Our analysis reveals that LLMs often fail to redirect problematic questions, even when the problematic premise is detected, and provide answers that could lead to suboptimal medical decision making. Our benchmark and results reveal a novel and substantial gap in how LLMs perform under the conditions of real-world health communication, highlighting critical safety concerns for patient-facing medical AI systems. Code and dataset are available at https://github.com/srsambara-1/MedRedFlag.

Related papers

• What Patients Really Ask: Exploring the Effect of False Assumptions in Patient Information Seeking [5.012718216094781]
  Patients are increasingly using large language models (LLMs) to seek answers to their healthcare-related questions.<n>We sourced data from Google's People Also Ask feature by querying the top 200 prescribed medications in the United States.<n>A considerable portion of the collected questions contains incorrect assumptions and dangerous intentions.
  arXiv  Detail & Related papers  (2026-01-22T05:56:14Z)
• Dr. Bias: Social Disparities in AI-Powered Medical Guidance [0.04297070083645049]
  Large Language Models (LLMs) are increasingly proving to be competitive.<n>We show that, when LLMs are used for medical advice generation, they generate responses that systematically differ between social groups.<n>In particular, Indigenous and intersex patients receive advice that is less readable and more complex.
  arXiv  Detail & Related papers  (2025-10-10T09:06:35Z)
• Dr. GPT Will See You Now, but Should It? Exploring the Benefits and Harms of Large Language Models in Medical Diagnosis using Crowdsourced Clinical Cases [7.894865736540358]
  Large Language Models (LLMs) are used in high-stakes applications such as medical (self-diagnosis) and preliminary triage.<n>This paper presents the findings from a university-level competition that leveraged a novel, crowdsourced approach for evaluating the effectiveness of LLMs.
  arXiv  Detail & Related papers  (2025-06-13T17:12:47Z)
• Structured Outputs Enable General-Purpose LLMs to be Medical Experts [50.02627258858336]
  Large language models (LLMs) often struggle with open-ended medical questions.<n>We propose a novel approach utilizing structured medical reasoning.<n>Our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models.
  arXiv  Detail & Related papers  (2025-03-05T05:24:55Z)
• Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments [108.55277188617035]
  Large language models (LLMs) have been widely adopted in various downstream task domains, but their abilities to directly recall and apply factual medical knowledge remains under-explored.<n>We introduce the Medical Knowledge Judgment dataset (MKJ), a dataset derived from the Unified Medical Language System (UMLS), a comprehensive repository of standardized vocabularies and knowledge graphs.<n>Through a binary classification framework, MKJ evaluates LLMs' grasp of fundamental medical facts by having them assess the validity of concise, one-hop statements.
  arXiv  Detail & Related papers  (2025-02-20T05:27:51Z)
• ELOQ: Resources for Enhancing LLM Detection of Out-of-Scope Questions [52.33835101586687]
  We study out-of-scope questions, where the retrieved document appears semantically similar to the question but lacks the necessary information to answer it.<n>We propose a guided hallucination-based approach ELOQ to automatically generate a diverse set of out-of-scope questions from post-cutoff documents.
  arXiv  Detail & Related papers  (2024-10-18T16:11:29Z)
• Are LLMs Aware that Some Questions are not Open-ended? [58.93124686141781]
  We study whether Large Language Models are aware that some questions have limited answers and need to respond more deterministically. The lack of question awareness in LLMs leads to two phenomena: (1) too casual to answer non-open-ended questions or (2) too boring to answer open-ended questions.
  arXiv  Detail & Related papers  (2024-10-01T06:07:00Z)
• MedHalu: Hallucinations in Responses to Healthcare Queries by Large Language Models [26.464489158584463]
  Large language models (LLMs) are starting to complement traditional information seeking mechanisms such as web search.<n>LLMs are prone to hallucinations, generating plausible yet factually incorrect or fabricated information.<n>This work conducts a pioneering study on hallucinations in LLM-generated responses to real-world healthcare queries from patients.
  arXiv  Detail & Related papers  (2024-09-29T00:09:01Z)
• Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models [70.19081534515371]
  Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks. They generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. We propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers.
  arXiv  Detail & Related papers  (2024-07-04T18:47:42Z)
• Can AI Relate: Testing Large Language Model Response for Mental Health Support [23.97212082563385]
  Large language models (LLMs) are already being piloted for clinical use in hospital systems like NYU Langone, Dana-Farber and the NHS. We develop an evaluation framework for determining whether LLM response is a viable and ethical path forward for the automation of mental health treatment.
  arXiv  Detail & Related papers  (2024-05-20T13:42:27Z)
• ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences [51.66185471742271]
  We propose ChiMed-GPT, a benchmark LLM designed explicitly for Chinese medical domain. ChiMed-GPT undergoes a comprehensive training regime with pre-training, SFT, and RLHF. We analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients.
  arXiv  Detail & Related papers  (2023-11-10T12:25:32Z)
• A Survey of Large Language Models in Medicine: Progress, Application, and Challenge [85.09998659355038]
  Large language models (LLMs) have received substantial attention due to their capabilities for understanding and generating human language. This review aims to provide a detailed overview of the development and deployment of LLMs in medicine.
  arXiv  Detail & Related papers  (2023-11-09T02:55:58Z)
• FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation [92.43001160060376]
  We study the factuality of large language models (LLMs) in the context of answering questions that test current world knowledge. We introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA.
  arXiv  Detail & Related papers  (2023-10-05T00:04:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.