LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It

• URL: http://arxiv.org/abs/2608.31016v1
• Date: Mon, 31 Aug 2026 15:59:54 GMT
• Title: LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It
• Abstract summary: Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission.<n>Our benchmark has 500 single rubric-error note pairs from audited fact sheets.<n>On single notes, no design flags omissions reliably more often than perfect notes.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the benchmark, prompts and judgements.

Related papers

• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints [4.41537705949485]
  Language-model judges now gate training data, score generations, and drive leaderboards.<n>We audited that assumption in two preregistered campaigns with every threshold fixed in advance.<n>Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.<n>We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist.
  arXiv  Detail & Related papers  (2026-09-03T17:59:43Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails [0.0]
  Self-improving agent pipelines have a problem at their center.<n>An evaluation signal failed in eleven classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking.<n>We describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply.
  arXiv  Detail & Related papers  (2026-09-02T07:54:23Z)
• One note in three: a verified census of three deployed AI scribes, and the instrument that counted it [0.0]
  We audited three commercial AI scribes on the same 142 consultations.<n>12 discovery passes proposed 13,678 candidate errors; 5,898 clearing an importance filter went to an adversarial panel of two models.<n>One note in three carries a verified failure, concentrated in allergy and medication information.
  arXiv  Detail & Related papers  (2026-08-31T16:00:27Z)
• Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit [48.811961095386]
  Hindsight Memory-PRM exploits this audit trail twice.<n> offline to train an operation-conditioned memory-utility critic.<n>Online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit.
  arXiv  Detail & Related papers  (2026-08-30T06:53:13Z)
• Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail [0.0]
  Defining the corrective gain of a failure record as the change in log-probability of re-emitting the action that just failed, we find the gain is negative for every instruction-tuned model.<n>The problem is in the harness, not the model's grasp of error messages, and that predicts which remedies work.
  arXiv  Detail & Related papers  (2026-08-24T10:19:17Z)
• FACTWASH: Catching AI Rewrites That Wash Hearsay into Fact [0.0]
  AI systems rewrite information constantly: conversations become stored memories, documents become answers.<n>We call that fact-washing, an open-source write-time gate that catches it deterministically, with named flags and evidence rather than an judge.<n>Building it answers a practical question: when does a cheap check suffice, and when do you need a model?
  arXiv  Detail & Related papers  (2026-08-04T09:20:54Z)
• Explaining AI-Image Detection: What the Heatmap Actually Shows [65.01025489527173]
  We build a detector and attach an attribution map as its evidence.<n>We measure what that pair delivers on 186,527 images under controls.<n>Whether an attribution ranking exists at all depends on whether the detector reacts to the image.
  arXiv  Detail & Related papers  (2026-07-31T16:07:05Z)
• HALLMARK: Diagnosing Three Failure Modes in LLM Citation Verifiers [30.02579334273979]
  GPTZero found 53 papers with hallucinated citations among NeurIPS 2025's accepted set.<n>No shared benchmark compares verifiers and gives detailed failure diagnostics.<n>Hallucination benchmark: 2,526 Bib entries spanning 14 types, three difficulty tiers, six diagnostic sub-tests per entry.
  arXiv  Detail & Related papers  (2026-07-20T13:21:27Z)
• Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Reviews [37.684628465527545]
  We operationalise Kahneman's dual-process theory into a structured rubric for peer review and release Kahneman4Review.<n>A benchmark of 3,563 rated reviews scored along nine theoretically motivated textual dimensions, eight bias diagnostics, and a continuous reasoning-quality score.
  arXiv  Detail & Related papers  (2026-07-12T00:03:42Z)
• Has This Checkpoint Been Abliterated? A Two-Signal Audit and Its Failure Map [0.676855875213031]
  We combine two cheap internal signals, a reference-anchored activation refusal-gap and a weight-recovery energy of the base-to-candidate weight difference, into a threshold-free checkpoint audit.<n>On a 273-checkpoint registry spanning Qwen, DeepSeek-distilled Qwen, Llama, and Gemma, our z-sum separates 57 public abliterations from 37 benign fine-tunes, merges, and instruction-tunes at AUROC 0.95.<n>The audit is effective triage, not tamper-proofing: it presumes an attested reference, and its claims are bounded by the registry we evaluate
  arXiv  Detail & Related papers  (2026-07-02T08:15:25Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• INSURE-Dial: A Phase-Aware Conversational Dataset & Benchmark for Compliance Verification and Phase Detection [40.157413213893165]
  Administrative phone tasks drain roughly 1 trillion USD annually from U.S. healthcare.<n>We introduce INSURE-Dial, the first public benchmark for developing and assessing compliance-aware voice agents.
  arXiv  Detail & Related papers  (2026-01-28T17:27:49Z)
• The Decisive Power of Indecision: Low-Variance Risk-Limiting Audits and Election Contestation via Marginal Mark Recording [51.82772358241505]
  Risk-limiting audits (RLAs) are techniques for verifying the outcomes of large elections. We define new families of audits that improve efficiency and offer advances in statistical power. New audits are enabled by revisiting the standard notion of a cast-vote record so that it can declare multiple possible mark interpretations.
  arXiv  Detail & Related papers  (2024-02-09T16:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.