Same Facts, Different Diagnosis: Measuring and Mitigating Narrative Anchoring in Clinical Language Models

• URL: http://arxiv.org/abs/2607.27384v1
• Date: Wed, 29 Jul 2026 18:43:47 GMT
• Title: Same Facts, Different Diagnosis: Measuring and Mitigating Narrative Anchoring in Clinical Language Models
• Abstract summary: We construct a dataset of 1,000 USMLE clinical vignettes, each rewritten into three sociolinguistically distinct personas.<n>Narrative Anchoring is statistically significant under direct prompting in every model tested, with a Narrative Anchoring Gap of 0.064 to 0.151.<n>We introduce NarrativeShield, a three-agent pipeline that structurally extracts and verifies clinical facts before diagnostic reasoning begins.
• Score: 0.15293427903448023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models used for clinical diagnostic reasoning are sensitive to sociolinguistic register, not just clinical content. We term this failure mode Narrative Anchoring: identical clinical facts expressed in different registers cause diagnostic outputs to diverge. Unlike prior demographic-bias work, which manipulates explicit identity tokens such as race or income, our benchmark isolates register as the sole channel of variation, with no demographic marker present in any form. We construct a dataset of 1,000 USMLE clinical vignettes, each rewritten into three sociolinguistically distinct personas under an independently audited fact-preservation guarantee, verified by a separate model that never sees the generation prompt. Across seven language models spanning three architecture families and scales, Narrative Anchoring is statistically significant under direct prompting in every model tested, with a Narrative Anchoring Gap of 0.064 to 0.151. Chain-of-thought reasoning and explicit debiasing instructions reduce the bias only partially, and their apparent gains are frequently confounded by accuracy collapse. We introduce NarrativeShield, a three-agent pipeline that structurally extracts and verifies clinical facts before diagnostic reasoning begins, reducing the Narrative Anchoring Gap to near-zero ($-0.004$ to $0.037$) and achieving the lowest rate of severely unstable decisions (DSS $<$ 0.8) of any method across all models, at a modest and mechanistically expected accuracy cost for most models. A stress test using a non-instruction-tuned base model shows that executing a debiasing intervention at all is gated by zero-shot instruction-following ability, not prompt content alone. We release our dataset, human-validated for fact preservation, as a standalone resource for studying register-based clinical bias.

Related papers

• Conformalized Large Language Models under Configuration Shift [46.502741608809174]
  Conformal prediction is a distribution-free framework for uncertainty quantification.<n>Nonconformity scores are often induced by an inference pipeline, not just a fixed model.<n>We show that configuration shift consistently erodes CP validity.
  arXiv  Detail & Related papers  (2026-08-02T19:34:41Z)
• RareLens: Towards End-to-End Rare Disease Care via Aligning Divergent Large Language Model Reasoning [42.80742579670377]
  RareLens is a system that supports clinical decision-making across the entire rare disease trajectory.<n>Developed and evaluated on RareBench, a real-world dataset of 157,525 cases spanning all 33 Orphanet categories and more than 7,000 conditions.<n>It achieved an area under the curve of 0.917 for screening and top-1 accuracies of 65.5% and 89.8% for diagnosis and treatment.
  arXiv  Detail & Related papers  (2026-07-25T16:58:10Z)
• The C-index illusion: discrimination without calibration in published survival models [0.0]
  We reproduce three published survival-ML models across three structurally distinct domains.<n>We test five pre-registered hypotheses under a Holm-corrected family-wise error rate.
  arXiv  Detail & Related papers  (2026-07-21T19:13:54Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation [0.0]
  We show that models possess the capability to detect fabricated statistics but do not recruit this capability during multi-source synthesis.<n>Specifically, source influence is governed by a methodology-register gate that responds to the distributional register of analytical text but not to numeric validity.
  arXiv  Detail & Related papers  (2026-06-03T20:15:48Z)
• What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models [0.0]
  Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter.<n>We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks.<n>We find three broad results. First, evaluation design can materially change the conclusion.<n>Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant.
  arXiv  Detail & Related papers  (2026-05-03T20:05:08Z)
• SAAG: Structured Agent Assessment and Grounding [13.62148061055744]
  We propose a cascaded diagnostic framework that decomposes agent-calling evaluation into three sequential stages.<n>We evaluate this framework on a benchmark derived from Glaive's function-calling dataset across registry sizes of 5, 10, and 15 agents.
  arXiv  Detail & Related papers  (2026-04-30T19:33:58Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Agentic Cognitive Profiling: Realigning Automated Alzheimer's Disease Detection with Clinical Construct Validity [66.94391219005291]
  This paper proposes Agentic Cognitive Profiling (ACP), an agentic framework that realigns automated screening with clinical protocol logic.<n>Central to our design is decoupling semantic understanding from measurement by delegating all quantification to deterministic function calling.<n>ACP achieves 90.5% score match rate in task examination and 85.3% accuracy in AD prediction, surpassing popular baselines.
  arXiv  Detail & Related papers  (2026-03-18T06:15:35Z)
• MeCaMIL: Causality-Aware Multiple Instance Learning for Fair and Interpretable Whole Slide Image Diagnosis [40.3028468133626]
  Multiple instance learning (MIL) has emerged as the dominant paradigm for whole slide image (WSI) analysis in computational pathology.<n>textbfMeCaMIL, a causality-aware MIL framework, explicitly models demographic confounders through structured causal graphs.<n>MeCaMIL achieves superior fairness -- demographic disparity variance drops by over 65% relative reduction on average across attributes.
  arXiv  Detail & Related papers  (2025-11-14T06:47:21Z)
• MEDEQUALQA: Evaluating Biases in LLMs with Counterfactual Reasoning [7.167933033102407]
  We introduce MEDEQUALQA, a counterfactual benchmark that perturbs only patient pronouns while holding critical symptoms and conditions constant.<n>We evaluate a GPT-4.1 model and compute Semantic Textual Similarity (STS) between reasoning traces to measure stability across pronoun variants.<n>Our results show overall high similarity (mean STS >0.80), but reveal consistent localized divergences in cited risk factors, guideline anchors, and differential ordering.
  arXiv  Detail & Related papers  (2025-10-09T22:12:58Z)
• Clinical Uncertainty Impacts Machine Learning Evaluations [40.773483049446426]
  We argue that machine-learning evaluations should explicitly account for annotation uncertainty using probabilistic metrics that directly operate on distributions.<n>We urge the community to release raw annotations for datasets and to adopt uncertainty-aware evaluation so that performance estimates may better reflect clinical data.
  arXiv  Detail & Related papers  (2025-09-26T11:56:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.