Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts

• URL: http://arxiv.org/abs/2608.15254v1
• Date: Sat, 15 Aug 2026 14:18:18 GMT
• Title: Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts
• Abstract summary: A one-sentence DEI prompt appended to a medical question leads models to add patient demographic attributes the question never stated.<n>Across 47 models, four medical benchmarks, and 376,000 responses scored by a validated model-judge pipeline, a single DEI prompt raises the injection rate from 0.7% to 33.1%.
• Score: 2.4171019220503402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical-AI guidance increasingly recommends prompting language models to reason with attention to diversity, equity, and inclusion (DEI). We measure a side effect that misrepresents patients: a one-sentence DEI prompt appended to a medical question leads models to add patient demographic attributes (race, socioeconomic status, sex) the question never stated, in effect rewriting who the patient is. We call this demographic injection. Across 47 models, four medical benchmarks, and 376,000 responses scored by a validated model-judge pipeline, a single DEI prompt raises the injection rate from 0.7% to 33.1% (47x) in all 47 of 47 models, attributable to the equity content rather than to added length (18x above a length-matched control; p=1.4x10^-14). Most added content is a general population statement that leaves the answer unchanged, but a smaller subset attaches an attribute to the specific patient or changes the selected option (0.25-2.4% of responses, 99.8% toward the incorrect option), where the invented demographic changes the answer the model recommends. Phrasing scales the effect from 14% to 56%. DEI prompts are just one example of a more general mechanism. Any instruction that nudges how a model reasons can make it add unrequested details, including details about the patient. Flagged outputs are treated as model errors under study, not clinical guidance.

Related papers

• DementiaCare-Bench: A Modality-Validated Video Benchmark [83.92703885588868]
  Dementia affects an estimated 57 million people worldwide.<n>For most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia.<n>We present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories.
  arXiv  Detail & Related papers  (2026-09-11T14:52:49Z)
• Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice [0.0]
  Patients increasingly ask large language model (LLM) assistants which doctor to see.<n>We report a prespecified randomized algorithm audit of what causally moves those recommendations.
  arXiv  Detail & Related papers  (2026-08-14T15:39:10Z)
• When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models [2.064612766965483]
  We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions)<n>Our experiments reveal several concerning findings.
  arXiv  Detail & Related papers  (2026-03-26T23:04:20Z)
• Explainable Admission-Level Predictive Modeling for Prolonged Hospital Stay in Elderly Populations: Challenges in Low- and Middle-Income Countries [65.4286079244589]
  Prolonged length of stay (pLoS) is a significant factor associated with the risk of adverse in-hospital events.<n>We develop and explain a predictive model for pLos using admission-level patient and hospital administrative data.
  arXiv  Detail & Related papers  (2026-01-07T23:35:24Z)
• MedVoiceBias: A Controlled Study of Audio LLM Behavior in Clinical Decision-Making [24.535496581050094]
  Large language models transition from text-based interfaces to audio interactions in clinical settings.<n>We evaluated these models on 170 clinical cases, each synthesized into speech from 36 distinct voice profiles.
  arXiv  Detail & Related papers  (2025-11-10T00:44:37Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• Personalized Prediction Models for Changes in Knee Pain among Patients with Osteoarthritis Participating in Supervised Exercise and Education [0.0]
  Knee osteoarthritis (OA) is a widespread chronic condition that impairs mobility and diminishes quality of life. Despite the proven benefits of exercise therapy and patient education in managing the OA symptoms pain and functional limitations, these strategies are often underutilized. To improve predictions, new variables beyond those in the GLA:D are required.
  arXiv  Detail & Related papers  (2024-10-16T14:15:01Z)
• Equity in Healthcare: Analyzing Disparities in Machine Learning Predictions of Diabetic Patient Readmissions [2.321323878201932]
  This study investigates how machine learning (ML) models can predict hospital readmissions for diabetic patients fairly and accurately across different demographics. We compared models like Deep Learning, Generalized Linear Models, Gradient Boosting Machines (GBM), and Naive Bayes. GBM stood out with an F1-score of 84.3% and accuracy of 82.2%, accurately predicting readmissions across demographics.
  arXiv  Detail & Related papers  (2024-03-27T23:49:22Z)
• Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models [49.95603725998561]
  We propose a new paradigm to build robust and interpretable medical image classifiers with natural language concepts. Specifically, we first query clinical concepts from GPT-4, then transform latent image features into explicit concepts with a vision-language model.
  arXiv  Detail & Related papers  (2023-10-04T21:57:09Z)
• Improving Selective Visual Question Answering by Learning from Your Peers [74.20167944693424]
  Visual Question Answering (VQA) models can have difficulties abstaining from answering when they are wrong. We propose Learning from Your Peers (LYP) approach for training multimodal selection functions for making abstention decisions. Our approach uses predictions from models trained on distinct subsets of the training data as targets for optimizing a Selective VQA model.
  arXiv  Detail & Related papers  (2023-06-14T21:22:01Z)
• What Do You See in this Patient? Behavioral Testing of Clinical NLP Models [69.09570726777817]
  We introduce an extendable testing framework that evaluates the behavior of clinical outcome models regarding changes of the input. We show that model behavior varies drastically even when fine-tuned on the same data and that allegedly best-performing models have not always learned the most medically plausible patterns.
  arXiv  Detail & Related papers  (2021-11-30T15:52:04Z)
• Classification supporting COVID-19 diagnostics based on patient survey data [82.41449972618423]
  logistic regression and XGBoost classifiers, that allow for effective screening of patients for COVID-19 were generated. The obtained classification models provided the basis for the DECODE service (decode.polsl.pl), which can serve as support in screening patients with COVID-19 disease. This data set consists of more than 3,000 examples is based on questionnaires collected at a hospital in Poland.
  arXiv  Detail & Related papers  (2020-11-24T17:44:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.