Evaluating the Presence of Sex Bias in Clinical Reasoning by Large Language Models

• URL: http://arxiv.org/abs/2602.04392v1
• Date: Wed, 04 Feb 2026 10:21:38 GMT
• Title: Evaluating the Presence of Sex Bias in Clinical Reasoning by Large Language Models
• Abstract summary: Large language models (LLMs) are increasingly embedded in healthcare for documentation, education, and clinical decision support.<n>We examined whether contemporary LLMs exhibit sex-specific biases in clinical reasoning and how model configuration influences these behaviours.
• Score: 0.5872014229110214
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are increasingly embedded in healthcare workflows for documentation, education, and clinical decision support. However, these systems are trained on large text corpora that encode existing biases, including sex disparities in diagnosis and treatment, raising concerns that such patterns may be reproduced or amplified. We systematically examined whether contemporary LLMs exhibit sex-specific biases in clinical reasoning and how model configuration influences these behaviours. We conducted three experiments using 50 clinician-authored vignettes spanning 44 specialties in which sex was non-informative to the initial diagnostic pathway. Four general-purpose LLMs (ChatGPT (gpt-4o-mini), Claude 3.7 Sonnet, Gemini 2.0 Flash and DeepSeekchat). All models demonstrated significant sex-assignment skew, with predicted sex differing by model. At temperature 0.5, ChatGPT assigned female sex in 70% of cases (95% CI 0.66-0.75), DeepSeek in 61% (0.57-0.65) and Claude in 59% (0.55-0.63), whereas Gemini showed a male skew, assigning a female sex in 36% of cases (0.32-0.41). Contemporary LLMs exhibit stable, model-specific sex biases in clinical reasoning. Permitting abstention reduces explicit labelling but does not eliminate downstream diagnostic differences. Safe clinical integration requires conservative and documented configuration, specialty-level clinical data auditing, and continued human oversight when deploying general-purpose models in healthcare settings.

Related papers

• CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning [37.91882261875461]
  Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably.<n>We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking.
  arXiv  Detail & Related papers  (2026-06-30T12:56:42Z)
• Gender-Dependent Diagnostic Substitution in LLM Medical Triage: Same Symptoms, Unequal Urgency [0.0]
  We investigate whether large language models produce different medical triage recommendations for identical neurological symptoms when only the patient's gender and age vary.<n>We present a standardized symptom profile across seven demographic conditions.<n>We find a stark, systemic gender-dependent triage disparity.
  arXiv  Detail & Related papers  (2026-06-02T13:35:12Z)
• EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triage [0.0]
  We present EQUITRIAGE, a fairness audit of large language models (LLMs) in emergency departments.<n>All five models produced flip rates above a pre-registered 5% threshold.<n>Group parity, counterfactual invariance, and gender calibration are distinct fairness properties.
  arXiv  Detail & Related papers  (2026-05-05T17:20:55Z)
• PsychBench: Auditing Epidemiological Fidelity in Large Language Model Mental Health Simulations [0.0]
  Large language models are increasingly deployed to simulate patients for clinical training, research, and mental health tools.<n>We introduce PsychBench, the first epidemiological audit of LLM patient simulation.<n>We show that models produce clinically plausible individuals while misrepresenting the populations they are drawn from.
  arXiv  Detail & Related papers  (2026-04-19T10:05:25Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations [60.2076951536797]
  Large Language Models (LLMs) are increasingly deployed in healthcare scenarios.<n>It is unclear to which extend LLMs could identify and adhere to clinical guidelines during conversations.<n>CPGBench is an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-03-26T09:00:55Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Smart Diagnosis and Early Intervention in PCOS: A Deep Learning Approach to Women's Reproductive Health [0.0]
  Polycystic Ovary Syndrome (PCOS) is a widespread disorder in women of reproductive age.<n>In this paper, we design a powerful framework based on transfer learning for classifying ovarian ultrasound images.<n>The model was trained on an online dataset containing 3856 ultrasound images of cyst-infected and non-infected patients.
  arXiv  Detail & Related papers  (2026-02-04T18:58:21Z)
• A DeepSeek-Powered AI System for Automated Chest Radiograph Interpretation in Clinical Practice [83.11942224668127]
  Janus-Pro-CXR (1B) is a chest X-ray interpretation system based on DeepSeek Janus-Pro model.<n>Our system outperforms state-of-the-art X-ray report generation models in automated report generation.
  arXiv  Detail & Related papers  (2025-12-23T13:26:13Z)
• A Women's Health Benchmark for Large Language Models [26.831715552066694]
  We introduce the Women's Health Benchmark (WHB), the first benchmark evaluating large language models (LLMs) performance specifically in women's health.<n>Our benchmark comprises 96 rigorously validated model stumps covering five medical specialties.<n>Current models show approximately 60% failure rates on the women's health benchmark, with performance varying dramatically across specialties and error types.
  arXiv  Detail & Related papers  (2025-12-18T19:44:28Z)
• MedVoiceBias: A Controlled Study of Audio LLM Behavior in Clinical Decision-Making [24.535496581050094]
  Large language models transition from text-based interfaces to audio interactions in clinical settings.<n>We evaluated these models on 170 clinical cases, each synthesized into speech from 36 distinct voice profiles.
  arXiv  Detail & Related papers  (2025-11-10T00:44:37Z)
• Evolving Diagnostic Agents in a Virtual Clinical Environment [75.59389103511559]
  We present a framework for training large language models (LLMs) as diagnostic agents with reinforcement learning.<n>Our method acquires diagnostic strategies through interactive exploration and outcome-based feedback.<n>DiagAgent significantly outperforms 10 state-of-the-art LLMs, including DeepSeek-v3 and GPT-4o.
  arXiv  Detail & Related papers  (2025-10-28T17:19:47Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)
• GeHirNet: A Gender-Aware Hierarchical Model for Voice Pathology Classification [4.504515306742906]
  We propose a novel framework that first identifies gender-specific pathological patterns using ResNet-50 on Mel spectrograms, then performs gender-conditioned disease classification.<n>This work advances voice pathology classification while reducing gender bias through hierarchical modeling of vocal characteristics.
  arXiv  Detail & Related papers  (2025-08-02T03:19:44Z)
• An Agentic System for Rare Disease Diagnosis with Traceable Reasoning [69.46279475491164]
  We introduce DeepRare, the first rare disease diagnosis agentic system powered by a large language model (LLM)<n>DeepRare generates ranked diagnostic hypotheses for rare diseases, each accompanied by a transparent chain of reasoning.<n>The system demonstrates exceptional diagnostic performance among 2,919 diseases, achieving 100% accuracy for 1013 diseases.
  arXiv  Detail & Related papers  (2025-06-25T13:42:26Z)
• Dataset Distribution Impacts Model Fairness: Single vs. Multi-Task Learning [2.9530211066840417]
  We evaluate the performance of skin lesion classification using ResNet-based CNNs.<n>We present a linear programming method for generating datasets with varying patient sex and class labels.
  arXiv  Detail & Related papers  (2024-07-24T15:23:26Z)
• What Do You See in this Patient? Behavioral Testing of Clinical NLP Models [69.09570726777817]
  We introduce an extendable testing framework that evaluates the behavior of clinical outcome models regarding changes of the input. We show that model behavior varies drastically even when fine-tuned on the same data and that allegedly best-performing models have not always learned the most medically plausible patterns.
  arXiv  Detail & Related papers  (2021-11-30T15:52:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.