One note in three: a verified census of three deployed AI scribes, and the instrument that counted it

• URL: http://arxiv.org/abs/2608.31017v1
• Date: Mon, 31 Aug 2026 16:00:27 GMT
• Title: One note in three: a verified census of three deployed AI scribes, and the instrument that counted it
• Abstract summary: We audited three commercial AI scribes on the same 142 consultations.<n>12 discovery passes proposed 13,678 candidate errors; 5,898 clearing an importance filter went to an adversarial panel of two models.<n>One note in three carries a verified failure, concentrated in allergy and medication information.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ambient AI scribes draft clinical notes under the reassurance that a clinician signs every note. We audited three commercial AI scribes on the same 142 consultations: 565 notes from recorded UK primary-care and US ambulatory encounters plus authored scenarios. Twelve discovery passes proposed 13,678 candidate errors; the 5,898 clearing an importance filter went to an adversarial panel of two models from different families, each told to refute what it could, and 618 survived. One note in three (31.3% [27.0, 35.6]) carries a verified failure, concentrated in allergy and medication information, invented patient identity, and history written up as examination on telephone consultations that can contain none. No product was given a patient record; setting aside the two classes a record would have prefilled, invented identity and dates, the rate is 24.8% [20.8, 29.0]. One failure mode did not fit our scheme, drawn from published scribe-error taxonomies: a treatment the clinician retracts, recorded as delivered care. Two clinicians adjudicated blind, disjoint samples: a physician author upheld 20 of 21 findings (95.2% [77.3, 99.2]) and an independent clinician, not an author, 12 of 12 ([75.8, 100]); both judged every sampled refusal genuine. A failure rate depends on the instrument as much as the scribes. With model, evidence and settings fixed, the review instruction alone moves the share of candidates verified from 9.3% to 79.0%, and the reviewing family moves it too: alone at that instruction the gentler flags 54.8% of notes against 27.8%. Between 28% and 97% of sampled notes carry a failure depending on the standard. Published audits disagree among themselves by a margin instrument differences alone can produce: omission is 54-86% of their errors against our 23.1%. We release all 618 findings with transcript-side evidence, every prompt and model version, and the re-runnable pipeline.

Related papers

• PaperDoctor: Evidence-Grounded and Actionable Feedback for Scientific Papers in Progress [130.58520199343707]
  PaperDoctor is an agent framework for pre-submission feedback with three key innovations.<n>First, a holistic hierarchical framework evaluates writing, layout, references, code, theory, prior work, and experiments.<n>Second, each finding contains an observation, a pointer to specific evidence such as a sentence, equation, or code line, and a revision suggestion.<n>Third, PaperDoctor selectively rebuilds and reruns experiments based on claim importance and compute budget.
  arXiv  Detail & Related papers  (2026-09-15T11:08:11Z)
• DementiaCare-Bench: A Modality-Validated Video Benchmark [83.92703885588868]
  Dementia affects an estimated 57 million people worldwide.<n>For most families the hardest part of care is not memory loss but the behavioral and psychological symptoms of dementia.<n>We present DementiaCare-Bench: 56 professionally produced caregiver training videos segmented into 94 clips across nine BPSD categories.
  arXiv  Detail & Related papers  (2026-09-11T14:52:49Z)
• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It [0.0]
  Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission.<n>Our benchmark has 500 single rubric-error note pairs from audited fact sheets.<n>On single notes, no design flags omissions reliably more often than perfect notes.
  arXiv  Detail & Related papers  (2026-08-31T15:59:54Z)
• SDoH-Aware Narrative Anchoring Bias in Medical LLMs for Trustworthy Clinical Decision Support [0.0]
  This paper evaluates the risk as SDoH aware narrative anchoring bias.<n>We use NarrativeShield SDoH MedQA, a counterfactual medical question answering dataset.<n>The final experiment uses 300 clinical cases and produces 8,100 model responses.
  arXiv  Detail & Related papers  (2026-08-24T04:58:32Z)
• Whose doctor does the AI recommend? An algorithm audit of reputation and demographic signals in large language model-assisted physician choice [0.0]
  Patients increasingly ask large language model (LLM) assistants which doctor to see.<n>We report a prespecified randomized algorithm audit of what causally moves those recommendations.
  arXiv  Detail & Related papers  (2026-08-14T15:39:10Z)
• Auditable agentic AI for evidence-grounded thyroid ultrasound diagnosis and reporting [59.709162055944915]
  ThyroidXAgent is an agentic AI system that coordinates specialized diagnostic tools and stores their outputs as an auditable case-level evidence record.<n>It was evaluated on 28,458 non-overlapping test cases, including 8,721 cases from 35 centres in the private NHC-MISD-TUS cohort.<n>It achieved a mean Dice score of 87.21 percent for segmentation and a mean AUROC of 0.9466 for benign-malignant classification.
  arXiv  Detail & Related papers  (2026-08-12T21:02:59Z)
• MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence [24.517280048376758]
  medvigil is a 300-case evaluation suite drawn from four public medical VQA sources.<n>Every gold answer, refusal option, candidate-answer set, paraphrase, false-premise trap, ROI box, and clinical risk tier is clinician-authored.<n>We report seven correctness-conditioned audit metrics that summarise into the medvigil Composite Score (MCS)<n>The independent radiologist scores MCS 83.3 at silent-failure rate 5.8%, leaving a 14.1-point composite headroom above the strongest audited model.
  arXiv  Detail & Related papers  (2026-05-08T15:55:30Z)
• Same Verdict, Different Reasons: LLM-as-a-Judge and Clinician Disagreement on Medical Chatbot Completeness [49.2667937337333]
  We stress-test this assumption for detecting incomplete patient-facing medical responses.<n>We evaluate three granularities (General-Likert, Analytical-Rubric, Dynamic-Checklist) and three backbone models across two clinician-annotated datasets.
  arXiv  Detail & Related papers  (2026-03-26T19:01:55Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• Evaluating Generative AI as an Educational Tool for Radiology Resident Report Drafting [1.5457333450799497]
  This study evaluates a HIPAA-compliant GPT-4o system that delivers automated feedback on breast imaging reports drafted by residents in real clinical settings.<n>Three common error types were identified: (1) omission or addition of key findings, (2) incorrect use or omission of technical descriptors, and (3) final assessment inconsistent with findings.<n>GPT-4o showed strong agreement with attending consensus: 90.5%, 78.3%, and 90.4% across error types.
  arXiv  Detail & Related papers  (2025-09-22T20:51:09Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• Impact of Large Language Model Assistance on Patients Reading Clinical Notes: A Mixed-Methods Study [46.5728291706842]
  We developed a patient-facing tool using large language models (LLMs) to make clinical notes more readable. We piloted the tool with clinical notes donated by patients with a history of breast cancer and synthetic notes from a clinician.
  arXiv  Detail & Related papers  (2024-01-17T23:14:52Z)
• Human Evaluation and Correlation with Automatic Metrics in Consultation Note Generation [56.25869366777579]
  In recent years, machine learning models have rapidly become better at generating clinical consultation notes. We present an extensive human evaluation study where 5 clinicians listen to 57 mock consultations, write their own notes, post-edit a number of automatically generated notes, and extract all the errors. We find that a simple, character-based Levenshtein distance metric performs on par if not better than common model-based metrics like BertScore.
  arXiv  Detail & Related papers  (2022-04-01T14:04:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.