IyawoBench v2.0: Extended Diagnostic Evaluation of Large Language Model Clinical Triage in Nigerian Primary Care

• URL: http://arxiv.org/abs/2607.29085v1
• Date: Fri, 31 Jul 2026 07:11:55 GMT
• Title: IyawoBench v2.0: Extended Diagnostic Evaluation of Large Language Model Clinical Triage in Nigerian Primary Care
• Abstract summary: We present IyawoBench v2.0, an extended diagnostic evaluation of large language model clinical triage.<n>We introduce a formal mathematical framework that decomposes triage safety into three distinct failure modes: Conservative Escalation Bias, Systematic Downgrade Bias, and Middle-Tier Instability.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models are being deployed as clinical triage tools in low and middle income countries where trained physicians are scarce. Existing safety metrics, however, produce misleading confidence: models scoring 100% on binary "did not send an emergency home" safety measures may nevertheless exhibit systematic failure modes that render them undeployable at scale. We present IyawoBench v2.0, an extended diagnostic evaluation of large language model clinical triage on 200 synthetic vignettes derived from 1,200 real patient encounters at 19 Nigerian primary health centres. We introduce a formal mathematical framework comprising fourteen definitions and two theorems that decompose triage safety into three distinct failure modes: Conservative Escalation Bias, Systematic Downgrade Bias, and Middle-Tier Instability. We propose the Escalation Bias Index and Expected Deployment Cost as novel metrics that expose failure modes hidden by conventional accuracy and sensitivity scores. Evaluated on three frontier models (Claude Sonnet 4.6, Llama 3.3 70B, Llama 3.1 8B) plus five naive baselines, we show that: (1) all three models exhibit at least one formal failure mode; (2) traditional sensitivity metrics conceal a 77 percentage point under-triage gap in Llama 3.1 8B; (3) the optimal model varies across three deployment scenarios (Emergency-Focused, System-Sustainability, Balanced), demonstrating that single-ranking benchmarks are inadequate for LMIC clinical AI selection. IyawoBench v2.0 provides both a rigorous benchmark and a diagnostic framework transferable to any triage-style clinical AI evaluation. All code, data, and analysis pipelines are publicly available.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• Closed-Loop Validation-Repair for Healthcare Interoperability: A Multi-Model Study of Schema Compliance in Clinical LLMs [0.0]
  We evaluate three open-source models, Qwen2.5 7B, Llama 3.1 8B, and Gemma2 9B, via local deployment across 320 clinical scenarios spanning ten medical specialties.<n> schema noncompliance is consistent across the three model families, with baseline compliance rates ranging from 85.9 to 91.6 percent.<n> validation-repair framework achieves 99.0 percent overall compliance, ranging from 98.4 to 99.4 percent across models.
  arXiv  Detail & Related papers  (2026-07-27T12:45:27Z)
• IyàwóBench: A Benchmark for Evaluating Large Language Model Clinical Triage Accuracy on Undifferentiated Febrile Illness in Nigerian Primary Health Settings [0.0]
  Undifferentiated febrile illness is the leading cause of primary care outpatient visits in Nigeria.<n>No validated benchmark exists for evaluating large language model (LLM) clinical triage reasoning.<n>We introduce IywBench v1.0, a dataset of 200 synthetic clinical vignettes across eight febrile illness categories.
  arXiv  Detail & Related papers  (2026-05-22T10:25:51Z)
• Adversarial Fragility and Language Vulnerability in Clinical AI: A Systematic Audit of Diagnostic Collapse Under Imperceptible Perturbations and Cross-Lingual Drift in Low-Resource Healthcare Settings [0.0]
  This study presents the first systematic dual audit of two safety vulnerabilities in clinical AI: adversarial image fragility and cross-lingual diagnostic drift.<n>Using DenseNet121, fine-tuned on the COVID-QU-Ex chest X-ray dataset, we demonstrate that diagnostic accuracy collapses from 89.3% to 62.0%.<n>In a parallel language experiment, we tested Llama3.1:8b and NatLAS (NLASAT) on 20 COVID-19 clinical cases presented in Standard English, Nigerian Pidgin (Naija), and Yoruba-inflected English.
  arXiv  Detail & Related papers  (2026-05-16T13:33:47Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context [82.32380418146656]
  Health-ORSC-Bench is the first large-scale benchmark designed to measure textbfOver-Refusal and textbfSafe Completion quality in healthcare.<n>Our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity.<n>Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants.
  arXiv  Detail & Related papers  (2026-01-25T01:28:52Z)
• ClinNet: Evidential Ordinal Regression with Bilateral Asymmetry and Prototype Memory for Knee Osteoarthritis Grading [3.337151338735509]
  Knee osteoarthritis (KOA) grading based on radiographic images is a critical yet challenging task.<n>In this work, we propose ClinNet, a novel trustworthy framework that addresses KOA grading as evidential ordinal regression problem.
  arXiv  Detail & Related papers  (2026-01-24T05:49:41Z)
• Diagnosing Hallucination Risk in AI Surgical Decision-Support: A Sequential Framework for Sequential Validation [5.469454486414467]
  Large language models (LLMs) offer transformative potential for clinical decision support in spine surgery.<n>LLMs pose significant risks through hallucinations, which are factually inconsistent or contextually misaligned outputs.<n>This study introduces a clinician-centered framework to quantify hallucination risks by evaluating diagnostic precision, recommendation quality, reasoning robustness, output coherence, and knowledge alignment.
  arXiv  Detail & Related papers  (2025-11-01T15:25:55Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.