PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage

• URL: http://arxiv.org/abs/2606.05463v2
• Date: Tue, 09 Jun 2026 17:31:07 GMT
• Title: PSEBench: A Controllable and Verifiable Benchmark for Evaluating LLMs in Patient Safety Event Triage
• Abstract summary: Patient safety event triage is a high-stakes task typically performed manually by patient safety experts.<n>We instantiate this method on Minnesota's 29 Reportable Adverse Health Events.<n> PSEBench is a 5,074-case benchmark with an agentic evaluation environment.
• Score: 11.521711012101102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Patient safety event triage, determining whether a clinical event is reportable under jurisdiction-specific policy, is a high-stakes task typically performed manually by patient safety experts. Although LLMs may support this workflow, reliable evaluation is limited by the lack of benchmarks to capture evidence-grounded policy reasoning, proactive information seeking for incomplete reports, and principled abstention in irreducibly ambiguous cases. We address this gap with a policy-grounded construction methodology centered on the clause card, a structured representation that factorizes regulatory text into auditable decision specifications. Combining clause cards with anchor-driven instantiation and closed-loop verification, our scalable pipeline produces narratives with by-construction ground truth and naturally supports generating missing information and uncertain variants. We instantiate this method on Minnesota's 29 Reportable Adverse Health Events, producing PSEBench, a 5,074-case benchmark with an agentic evaluation environment. Evaluation on 15 representative LLMs reveals consistent capability trends, demonstrates the benchmark's utility, and identifies actionable gaps toward reliable LLM-based patient safety event triage.

Related papers

• AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation [48.55515767840701]
  "AI slop" is artifacts, hallucinated vulnerabilities, plausible but incorrect patches, and semantically repackaged bug reports.<n>This paper surveys the empirical evidence, identifies a unifying mechanism, and traces a path toward trustworthy triage.
  arXiv  Detail & Related papers  (2026-08-26T11:48:38Z)
• Using Fine-Tuned LLMs to Identify Indicators of Vulnerability in UK Police Incident Logs [0.0]
  We explore whether an LLM-based classification pipeline can be adapted to estimate the prevalence of four vulnerability indicators in UK police incident narratives.<n>Mental ill health indicators are present in approximately one in five incidents, with lower prevalence for other indicators.<n> Correcting these biases required substantial human input and statistical adjustment, leaving considerable uncertainty.
  arXiv  Detail & Related papers  (2026-07-20T18:58:33Z)
• Steering LLM Viewpoints through Fabricated Evidence Injection [61.22850755490993]
  We introduce a two-phase attack framework that first repackages misleading statements with fabricated rationales, then instruct target LLMs to incorporate these viewpoints when responding to relevant queries.<n>Experiments on BBQ, ToxiGen, and our specialized dataset reveal that commercial LLMs without external safety classifiers remain highly vulnerable.<n>Building on this, we explore multiple defense strategies, among which a tailored safety policy enablesgpt-oss-safeguard to achieve 81% detection rate.
  arXiv  Detail & Related papers  (2026-06-04T14:48:23Z)
• Neuro-Symbolic Verification of LLM Outputs for Data-Sensitive Domains (extended preprint) [0.08594140167290099]
  This paper presents a hybrid verification architecture combining formal symbolic methods with neural semantic analysis.<n>For output validation, embedding-based semantic similarity detects contextual hallucinations.<n> Evaluation shows hallucination detection rates of over 83% for structured entities and 72% for semantic fabrications.
  arXiv  Detail & Related papers  (2026-05-26T12:32:17Z)
• Anonymous-by-Construction: An LLM-Driven Framework for Privacy-Preserving Text [0.42821598129654453]
  We develop an on-premise, LLM-driven substitution pipeline that anonymizes text.<n>We measure privacy, semantic utility, and trainability under privacy.<n>Our method attains state-of-the-art privacy, minimal topical drift, strong factual utility, and low trainability loss.
  arXiv  Detail & Related papers  (2026-03-17T23:46:15Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking [64.97768177044355]
  Large Language Models (LLMs) are increasingly deployed in real-world fact-checking systems.<n>We present FactArena, a fully automated arena-style evaluation framework.<n>Our analyses reveal significant discrepancies between static claim-verification accuracy and end-to-end fact-checking competence.
  arXiv  Detail & Related papers  (2026-01-06T02:51:56Z)
• Evaluating Metrics for Safety with LLM-as-Judges [1.93892819796757]
  This paper argues that although we cannot get deterministic evaluations from many natural language processing tasks, by adopting a basket of weighted metrics it may be possible to lower the risk of errors within an evaluation.
  arXiv  Detail & Related papers  (2025-12-17T17:24:49Z)
• MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them [52.764019220214344]
  Hallucinations pose critical risks for large language model (LLM)-based agents.<n>We present MIRAGE-Bench, the first unified benchmark for eliciting and evaluating hallucinations in interactive environments.
  arXiv  Detail & Related papers  (2025-07-28T17:38:29Z)
• Towards Safety Reasoning in LLMs: AI-agentic Deliberation for Policy-embedded CoT Data Creation [70.62656296780074]
  We propose AIDSAFE: Agentic Iterative Deliberation for Safety Reasoning, a novel data generation recipe.<n>A data refiner stage in AIDSAFE ensures high-quality outputs by eliminating repetitive, redundant, and deceptive thoughts.<n>Our evaluations demonstrate that AIDSAFE-generated CoTs achieve superior policy adherence and reasoning quality.
  arXiv  Detail & Related papers  (2025-05-27T21:34:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.