Truth, Trust, and Trouble: Medical AI on the Edge

• URL: http://arxiv.org/abs/2507.02983v2
• Date: Thu, 09 Oct 2025 02:17:59 GMT
• Title: Truth, Trust, and Trouble: Medical AI on the Edge
• Abstract summary: We present a benchmarking framework using a dataset of over 1,000 health questions.<n>We assess model performance across honesty, helpfulness, and harmlessness.<n>Few-shot prompting improves accuracy from 78% to 85%, and all models show reduced helpfulness on complex queries.
• Score: 16.87709061954049
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) hold significant promise for transforming digital health by enabling automated medical question answering. However, ensuring these models meet critical industry standards for factual accuracy, usefulness, and safety remains a challenge, especially for open-source solutions. We present a rigorous benchmarking framework using a dataset of over 1,000 health questions. We assess model performance across honesty, helpfulness, and harmlessness. Our results highlight trade-offs between factual reliability and safety among evaluated models -- Mistral-7B, BioMistral-7B-DARE, and AlpaCare-13B. AlpaCare-13B achieves the highest accuracy (91.7%) and harmlessness (0.92), while domain-specific tuning in BioMistral-7B-DARE boosts safety (0.90) despite its smaller scale. Few-shot prompting improves accuracy from 78% to 85%, and all models show reduced helpfulness on complex queries, highlighting ongoing challenges in clinical QA.

Related papers

• MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoning [7.000170880015254]
  We propose MedPRMBench, the first process-level reward model benchmark for the medical domain.<n>It is built through a three-phase pipeline based on Clinical Reasoning Blueprints (CRBs)<n>The benchmark comprises 6,500 questions with 13,000 reasoning chains and 113,910 step-level labels, plus 6,879 questions for training.
  arXiv  Detail & Related papers  (2026-04-19T06:44:07Z)
• A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Models [34.38896321208398]
  Large language models (LLMs) are increasingly deployed across healthcare, yet existing benchmarks fail to capture model behavior under adversarial or ethically complex conditions.<n>We developed a multi-domain red teaming framework evaluating eleven contemporary LLMs across 690 clinically grounded scenarios.<n>Results revealed substantial performance variance, with mean scores ranging from 0.791 to 0.984.
  arXiv  Detail & Related papers  (2026-04-15T14:57:55Z)
• WHBench: Evaluating Frontier LLMs with Expert-in-the-Loop Validation on Women's Health Topics [0.6093095878992784]
  Women's Health Benchmark is an evaluation suite of 47 expert-crafted scenarios across 10 women's health topics.<n>We evaluate 22 models using a 23-criterion rubric spanning clinical accuracy, completeness, safety, communication quality, instruction following, equity, uncertainty handling, and guideline adherence.
  arXiv  Detail & Related papers  (2026-03-11T06:21:37Z)
• Robust Pre-Training of Medical Vision-and-Language Models with Domain-Invariant Multi-Modal Masked Reconstruction [0.0]
  We propose Robust Multi-Modal Masked Reconstruction (Robust-MMR), a self-supervised pre-training framework that explicitly incorporates objectives into masked vision-language learning.<n>We evaluate Robust-MMR on multiple medical vision-language benchmarks, including medical visual question answering (VQA-RAD), cross-domain image-text classification (MELINDA), and robust image-caption retrieval (ROCO)<n>Our results show that explicitly modeling robustness during pre-training leads to more reliable and transferable medical vision-language representations for real-world deployment.
  arXiv  Detail & Related papers  (2026-02-06T01:20:56Z)
• Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context [82.32380418146656]
  Health-ORSC-Bench is the first large-scale benchmark designed to measure textbfOver-Refusal and textbfSafe Completion quality in healthcare.<n>Our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity.<n>Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants.
  arXiv  Detail & Related papers  (2026-01-25T01:28:52Z)
• Cross-Platform Evaluation of Large Language Model Safety in Pediatric Consultations: Evolution of Adversarial Robustness and the Scale Paradox [0.0]
  Large language models (LLMs) are increasingly deployed in medical consultations, yet their safety under realistic user pressures remains understudied.<n>This study evaluated safety in consultations across models and platforms.<n>Findings show safety depends on alignment and architecture over scale, with smaller models outperforming larger.
  arXiv  Detail & Related papers  (2025-12-26T13:47:42Z)
• An Explainable Hybrid AI Framework for Enhanced Tuberculosis and Symptom Detection [55.35661671061754]
  Tuberculosis remains a critical global health issue, particularly in resource-limited and remote areas.<n>We propose a framework which enhances disease and symptom detection on chest X-rays by integrating two supervised heads and a self-supervised head.<n>Our model achieves an accuracy of 98.85% for distinguishing between COVID-19, tuberculosis, and normal cases, and a macro-F1 score of 90.09% for multilabel symptom detection.
  arXiv  Detail & Related papers  (2025-10-21T17:18:55Z)
• MedOmni-45°: A Safety-Performance Benchmark for Reasoning-Oriented LLMs in Medicine [69.08855631283829]
  We introduce Med Omni-45 Degrees, a benchmark designed to quantify safety-performance trade-offs under manipulative hint conditions.<n>It contains 1,804 reasoning-focused medical questions across six specialties and three task types, including 500 from MedMCQA.<n>Results show a consistent safety-performance trade-off, with no model surpassing the diagonal.
  arXiv  Detail & Related papers  (2025-08-22T08:38:16Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)
• Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation [69.63626052852153]
  We propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems.<n>We also introduce a dataset of 100 indirect harm scenarios, testing models' ability to foresee adverse, non-obvious outcomes from seemingly harmless user prompts.
  arXiv  Detail & Related papers  (2025-06-26T02:28:58Z)
• Benchmarking Open-Source Large Language Models on Healthcare Text Classification Tasks [2.7729041396205014]
  This study evaluates the classification performance of five open-source large language models (LLMs)<n>We report precision, recall, and F1 scores with 95% confidence intervals for all model-task combinations.
  arXiv  Detail & Related papers  (2025-03-19T12:51:52Z)
• Benchmarking Chinese Medical LLMs: A Medbench-based Analysis of Performance Gaps and Hierarchical Optimization Strategies [11.0505830548286]
  This study introduces a granular error taxonomy through systematic analysis of top 10 models on MedBench.<n> Evaluation of 10 leading models reveals vulnerabilities, despite achieving 0.86 accuracy in medical knowledge recall.<n>Our analysis uncovers systemic weaknesses in knowledge boundary enforcement and multi-step reasoning.
  arXiv  Detail & Related papers  (2025-03-10T13:28:25Z)
• Structured Outputs Enable General-Purpose LLMs to be Medical Experts [50.02627258858336]
  Large language models (LLMs) often struggle with open-ended medical questions.<n>We propose a novel approach utilizing structured medical reasoning.<n>Our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models.
  arXiv  Detail & Related papers  (2025-03-05T05:24:55Z)
• Safeguarding AI in Medical Imaging: Post-Hoc Out-of-Distribution Detection with Normalizing Flows [3.3968168503957625]
  Current OOD detection methods demand impractical retraining or modifications to pre-trained models.<n>We propose a post-hoc normalizing flow-based approach that seamlessly integrates with existing pre-trained models.<n>Our method achieved an AUROC of 84.61%, outperforming state-of-the-art methods like ViM (80.65%) and MDS (80.87%)
  arXiv  Detail & Related papers  (2025-02-17T10:31:24Z)
• LabSafety Bench: Benchmarking LLMs on Safety Issues in Scientific Labs [78.99703366417661]
  Large language models (LLMs) increasingly assist in tasks ranging from procedural guidance to autonomous experiment orchestration.<n>Such overreliance is particularly dangerous in high-stakes laboratory settings, where failures in hazard identification or risk assessment can result in severe accidents.<n>We propose the Laboratory Safety Benchmark (LabSafety Bench) to evaluate models on their ability to identify potential hazards, assess risks, and predict the consequences of unsafe actions in lab environments.
  arXiv  Detail & Related papers  (2024-10-18T05:21:05Z)
• Detecting Bias and Enhancing Diagnostic Accuracy in Large Language Models for Healthcare [0.2302001830524133]
  Biased AI-generated medical advice and misdiagnoses can jeopardize patient safety. This study introduces new resources designed to promote ethical and precise AI in healthcare.
  arXiv  Detail & Related papers  (2024-10-09T06:00:05Z)
• Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks [17.40940406100025]
  We introduce Meerkat, a new family of medical AI systems ranging from 7 to 70 billion parameters. Our systems achieved remarkable accuracy across six medical benchmarks. Meerkat-70B correctly diagnosed 21 out of 38 complex clinical cases, outperforming humans' 13.8.
  arXiv  Detail & Related papers  (2024-03-30T14:09:00Z)
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models [53.859446823312126]
  SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks.
  arXiv  Detail & Related papers  (2023-10-11T15:56:00Z)
• Conservative Prediction via Data-Driven Confidence Minimization [70.93946578046003]
  In safety-critical applications of machine learning, it is often desirable for a model to be conservative. We propose the Data-Driven Confidence Minimization framework, which minimizes confidence on an uncertainty dataset.
  arXiv  Detail & Related papers  (2023-06-08T07:05:36Z)
• UNITE: Uncertainty-based Health Risk Prediction Leveraging Multi-sourced Data [81.00385374948125]
  We present UNcertaInTy-based hEalth risk prediction (UNITE) model. UNITE provides accurate disease risk prediction and uncertainty estimation leveraging multi-sourced health data. We evaluate UNITE on real-world disease risk prediction tasks: nonalcoholic fatty liver disease (NASH) and Alzheimer's disease (AD) UNITE achieves up to 0.841 in F1 score for AD detection, up to 0.609 in PR-AUC for NASH detection, and outperforms various state-of-the-art baselines by up to $19%$ over the best baseline.
  arXiv  Detail & Related papers  (2020-10-22T02:28:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.