Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice

• URL: http://arxiv.org/abs/2608.14399v1
• Date: Fri, 14 Aug 2026 15:39:10 GMT
• Title: Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice
• Abstract summary: Patients increasingly ask large language model (LLM) assistants which doctor to see.<n>We report a prespecified randomized algorithm audit of what causally moves those recommendations.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Patients increasingly ask large language model (LLM) assistants which doctor to see, making these systems AI infomediaries: algorithms that intermediate one person's choice among other people and thereby decide, silently and at scale, which physicians become visible. We report a prespecified randomized algorithm audit of what causally moves those recommendations. Seven models (six open-weight; gpt-4o-mini) each chose among five synthetic family-medicine physician cards whose attributes were independently randomized across 3,024 choice sets, three patient personas, nine prompt paraphrases and nine experimental arms, yielding 40,068 scored responses; gender and ethnicity were signaled through names following correspondence-audit methodology. Reputation signals dominate: raising a rating from 3.9 to 4.7 increases choice probability by 31.4 percentage points (pp), and raising the fee from $90 to $190 lowers it by 20.0 pp. Demographic parity is rejected, but not in the direction human audit studies predict: female-signaled names gain 2.5 pp, and Hispanic-, South-Asian- and Black-signaled names gain 1.3-2.9 pp over White-signaled names, tilts worth $7-$14 per visit in fee-equivalent terms, and a content-free first-listed position is worth $11. Yet models mentioned gender or ethnicity in at most 0.03% of their stated reasons and abstained in 0.39% of trials, so these effects are invisible in the models' own explanations, and transparency obligations relying on model self-report would not detect them. One reasoning model failed the prespecified auditability gate outright. The frozen design makes the audit repeatable: any new model can be assessed against identical stimuli, making recurring behavioural audit, rather than self-reported explanation, the monitoring technology fit for purpose.

Related papers

• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• One note in three: a verified census of three deployed AI scribes, and the instrument that counted it [0.0]
  We audited three commercial AI scribes on the same 142 consultations.<n>12 discovery passes proposed 13,678 candidate errors; 5,898 clearing an importance filter went to an adversarial panel of two models.<n>One note in three carries a verified failure, concentrated in allergy and medication information.
  arXiv  Detail & Related papers  (2026-08-31T16:00:27Z)
• Demographic Injection in Medical Language Models under Diversity, Equity, and Inclusion Prompts [2.4171019220503402]
  A one-sentence DEI prompt appended to a medical question leads models to add patient demographic attributes the question never stated.<n>Across 47 models, four medical benchmarks, and 376,000 responses scored by a validated model-judge pipeline, a single DEI prompt raises the injection rate from 0.7% to 33.1%.
  arXiv  Detail & Related papers  (2026-08-15T14:18:18Z)
• RareLens: Towards End-to-End Rare Disease Care via Aligning Divergent Large Language Model Reasoning [42.80742579670377]
  RareLens is a system that supports clinical decision-making across the entire rare disease trajectory.<n>Developed and evaluated on RareBench, a real-world dataset of 157,525 cases spanning all 33 Orphanet categories and more than 7,000 conditions.<n>It achieved an area under the curve of 0.917 for screening and top-1 accuracies of 65.5% and 89.8% for diagnosis and treatment.
  arXiv  Detail & Related papers  (2026-07-25T16:58:10Z)
• EQUITRIAGE: A Fairness Audit of Gender Bias in LLM-Based Emergency Department Triage [0.0]
  We present EQUITRIAGE, a fairness audit of large language models (LLMs) in emergency departments.<n>All five models produced flip rates above a pre-registered 5% threshold.<n>Group parity, counterfactual invariance, and gender calibration are distinct fairness properties.
  arXiv  Detail & Related papers  (2026-05-05T17:20:55Z)
• Evaluation format, not model capability, drives triage failure in the assessment of consumer health AI [3.910019733981544]
  ChatGPT Health under-triages 51.6% of emergencies, concluding that consumer-facing AI triage poses safety risks.<n>We tested five frontier LLMs (GPT-5.2, Claude Sonnet 4.6, Claude Opus 4.6, Gemini 3 Flash, Gemini 3.1 Pro) on a 17-scenario partial replication bank.<n>The headline under-triage rate is highly contingent on evaluation format and should not be interpreted as a stable estimate of deployed triage behavior.
  arXiv  Detail & Related papers  (2026-03-12T00:58:22Z)
• From Black Box to Glass Box: Cross-Model ASR Disagreement to Prioto Review in Ambient AI Scribe Documentation [43.148402136307716]
  Cross-model disagreement among heterogeneous ASR systems can act as a reference-free uncertainty signal.<n>We transcribed 50 publicly available medical audio clips with eight ASR systems spanning commercial APIs and open-source engines.<n>Low-agreement regions were enriched for content disagreements, with the content fraction increasing from 53.9% to 73.9% across quintiles of high-risk mass.
  arXiv  Detail & Related papers  (2026-03-02T13:02:13Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Self-Transparency Failures in Expert-Persona LLMs: A Large-Scale Behavioral Audit [0.0]
  This study examines whether models exhibit self-transparency when assigned professional personas in high-stakes domains.<n>A Financial Advisor persona elicited 30.8% disclosure at the first prompt, while a Neurosurgeon persona elicited only 3.5%.<n>This creates the preconditions for a hypothesized Reverse Gell-Mann Amnesia effect, where appropriate disclosure in some domains leads users to overgeneralize trust to high-stakes contexts.
  arXiv  Detail & Related papers  (2025-11-26T16:41:49Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation [70.27452774899189]
  Large language models (LLMs) should signal low confidence on examples where they are incorrect, instead of misleading the user. As of November 2023, state-of-the-art LLMs do not provide access to these probabilities. Our best method composing linguistic confidences and surrogate model probabilities gives state-of-the-art confidence estimates on all 12 datasets.
  arXiv  Detail & Related papers  (2023-11-15T11:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.