IyàwóBench: A Benchmark for Evaluating Large Language Model Clinical Triage Accuracy on Undifferentiated Febrile Illness in Nigerian Primary Health Settings

• URL: http://arxiv.org/abs/2605.23465v1
• Date: Fri, 22 May 2026 10:25:51 GMT
• Title: IyàwóBench: A Benchmark for Evaluating Large Language Model Clinical Triage Accuracy on Undifferentiated Febrile Illness in Nigerian Primary Health Settings
• Abstract summary: Undifferentiated febrile illness is the leading cause of primary care outpatient visits in Nigeria.<n>No validated benchmark exists for evaluating large language model (LLM) clinical triage reasoning.<n>We introduce IywBench v1.0, a dataset of 200 synthetic clinical vignettes across eight febrile illness categories.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background. Undifferentiated febrile illness is the leading cause of primary care outpatient visits in Nigeria, yet no validated benchmark exists for evaluating large language model (LLM) clinical triage reasoning in West African primary health settings. Methods. We introduce IyàwóBench v1.0, a dataset of 200 synthetic clinical vignettes across eight febrile illness categories derived from statistical distributions of 1,200 real patient encounters at 19 primary health centres (PHCs) in Oyo State, Nigeria. Six LLMs were evaluated on structured triage classification across two metrics: triage accuracy and safety score. Results. All six models achieved 100% safety scores (95% CI: 96.4-100.0%), never downgrading a critical REFER NOW case to TREAT HERE. Triage accuracy varied substantially: Claude Sonnet (claude-sonnet-4-5) 67.5% (95% CI: 60.8-73.7%), Llama 4 Scout 59.5% (52.5-66.2%), Llama 3.3 70B 43.0% (36.2-50.0%), and Llama 3.1 8B 39.0% (32.4-45.9%). Two models demonstrated near-zero accuracy attributable to structured output non-compliance. Conclusions. Modern LLMs exhibit safe triage behaviour but vary substantially in structured clinical accuracy. Clinically engineered systems with embedded WHO guidelines outperform general-purpose models by up to 28.5 percentage points. IyàwóBench provides the first reproducible evaluation framework for LLM clinical decision support in West African primary care.

Related papers

• IyawoBench v2.0: Extended Diagnostic Evaluation of Large Language Model Clinical Triage in Nigerian Primary Care [0.0]
  We present IyawoBench v2.0, an extended diagnostic evaluation of large language model clinical triage.<n>We introduce a formal mathematical framework that decomposes triage safety into three distinct failure modes: Conservative Escalation Bias, Systematic Downgrade Bias, and Middle-Tier Instability.
  arXiv  Detail & Related papers  (2026-07-31T07:11:55Z)
• MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation [0.49454229616760287]
  Large language models achieve strong scores on medical benchmarks, yet these benchmarks evaluate each question in isolation.<n>We introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI.<n>We propose Evidence-Anchored RAG, a three-stage retrieval method that replaces aggregate similarity with an evidence coverage objective.
  arXiv  Detail & Related papers  (2026-07-15T21:56:19Z)
• CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning [37.91882261875461]
  Large Language Models (LLMs) achieve strong results on many medical benchmarks, but their clinical reasoning remains difficult to evaluate reliably.<n>We introduce CLExEval, a human-in-the-loop framework for evaluating LLM clinical reasoning under progressive information masking.
  arXiv  Detail & Related papers  (2026-06-30T12:56:42Z)
• Adversarial Fragility and Language Vulnerability in Clinical AI: A Systematic Audit of Diagnostic Collapse Under Imperceptible Perturbations and Cross-Lingual Drift in Low-Resource Healthcare Settings [0.0]
  This study presents the first systematic dual audit of two safety vulnerabilities in clinical AI: adversarial image fragility and cross-lingual diagnostic drift.<n>Using DenseNet121, fine-tuned on the COVID-QU-Ex chest X-ray dataset, we demonstrate that diagnostic accuracy collapses from 89.3% to 62.0%.<n>In a parallel language experiment, we tested Llama3.1:8b and NatLAS (NLASAT) on 20 COVID-19 clinical cases presented in Standard English, Nigerian Pidgin (Naija), and Yoruba-inflected English.
  arXiv  Detail & Related papers  (2026-05-16T13:33:47Z)
• A Breast Vision Pathology Foundation Model for Real-world Clinical Utility [65.57568187389113]
  We present bfBRAVE, a breast-adaptive pathology foundation model developed and evaluated using a total resource of 101,638 breast wholeslide images.<n>BRAVE supported practical roles in the clinical workflow, including safe exclusion of low-risk cases from routine AI-assisted second-review rescue of initially missed positives and prioritization of cases for further assessment.
  arXiv  Detail & Related papers  (2026-05-06T07:44:39Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations [60.2076951536797]
  Large Language Models (LLMs) are increasingly deployed in healthcare scenarios.<n>It is unclear to which extend LLMs could identify and adhere to clinical guidelines during conversations.<n>CPGBench is an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-03-26T09:00:55Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• LiveClin: A Live Clinical Benchmark without Leakage [50.45415584327275]
  LiveClin is a live benchmark designed for approximating real-world clinical practice.<n>We transform authentic patient cases into complex, multimodal evaluation scenarios that span the entire clinical pathway.<n>Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%.
  arXiv  Detail & Related papers  (2026-02-18T03:59:46Z)
• COPE: Chain-Of-Thought Prediction Engine for Open-Source Large Language Model Based Stroke Outcome Prediction from Clinical Notes [23.044580867637105]
  Chain-of-Thought (CoT) Outcome Prediction Engine (COPE) is a reasoning-enhanced large language model framework for predicting outcomes from unstructured clinical notes.<n>This study included 464 acute ischemic stroke (AIS) patients with discharge summaries and 90-day modified Rankin Scale (mRS) scores.<n>COPE achieved an MAE of 1.01 ( 95% CI 0.92-1.11), +/-1 accuracy of 74.4% (69.9, 78.8%), and exact accuracy of 32.8% (28.0, 37.6%).
  arXiv  Detail & Related papers  (2025-12-02T07:44:20Z)
• MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks [47.486705282473984]
  Large language models (LLMs) achieve near-perfect scores on medical exams.<n>These evaluations inadequately reflect complexity and diversity of real-world clinical practice.<n>We introduce MedHELM, an evaluation framework for assessing LLM performance for medical tasks.
  arXiv  Detail & Related papers  (2025-05-26T22:55:49Z)
• Utilizing Machine Learning Models to Predict Acute Kidney Injury in Septic Patients from MIMIC-III Database [0.0]
  Sepsis is a severe condition that causes the body to respond incorrectly to an infection.<n>For septic patients, approximately 50% develop acute kidney injury (AKI)<n>Models that can accurately predict AKI based on specific qualities of septic patients are crucial for early detection and intervention.
  arXiv  Detail & Related papers  (2024-12-04T22:05:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.