Deliberative multi-agent large language models improve clinical reasoning in ophthalmology

• URL: http://arxiv.org/abs/2603.21447v1
• Date: Sun, 22 Mar 2026 23:36:48 GMT
• Title: Deliberative multi-agent large language models improve clinical reasoning in ophthalmology
• Abstract summary: Large language models (LLMs) show potential for ophthalmic clinical reasoning, yet individual models risk introducing harm.<n>We evaluated whether multi-agent LLM deliberative councils improve diagnostic performance and mitigate harm.<n>Councils consistently outperformed pooled individual models across all three tiers.
• Score: 6.0314253430783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) show potential for ophthalmic clinical reasoning, yet individual models risk introducing harm. We evaluated whether multi-agent LLM deliberative councils improve diagnostic performance and mitigate harm compared to individual LLMs. In a comparative cross-sectional study, we assessed 12 individual LLMs and three multi-agent councils on 100 ophthalmology clinical vignettes. Each council comprised four models assembled by type: proprietary flagship, proprietary fast, and open-source. Models independently answered a vignette, anonymously ranked one another's responses, and a designated chair synthesized all responses and peer reviews into a final answer. Councils consistently outperformed pooled individual models across all three tiers. Accuracy improved for proprietary flagship (95.0% vs 90.8%; risk difference [RD]: 4.25 [95% CI: 0.45, 8.05]), proprietary fast (96.0% vs 86.5%; RD: 9.50 [5.31, 13.59]), and open-source councils (91.0% vs 83.2%; RD: 7.75 [4.17, 11.33]). Harm rates declined for proprietary flagship (10.0% vs 22.5%; RD: -12.50 [-16.86, -8.14]), proprietary fast (16.0% vs 31.8%; RD: -15.75 [-21.49, -10.01]), and open-source councils (22.0% vs 38.5%; RD: -16.50 [-22.27, -10.73]). Coverage analysis revealed net positive gains for accuracy (ΔCoverage: 4.4-9.8 percentage points) and safety (ΔCoverage: 13.6-20.6), indicating councils recovered correct diagnoses and averted harm. Councils elevated correct diagnoses to higher rank positions; and produced more complete differentials and management plans (all P<.05). Harmful council responses showed reduced combined commission-and-omission errors and tended to be less severe. Structured deliberation via multi-agent LLM councils may enhance the reliability of LLM-assisted ophthalmic clinical reasoning.

Related papers

• Preserving Disagreement: Architectural Heterogeneity and Coherence Validation in Multi-Agent Policy Simulation [0.0]
  Multi-agent deliberation systems using large language models (LLMs) are increasingly proposed for policy simulation.<n>They suffer from artificial consensus: evaluator agents converge on the same option regardless of their assigned value perspectives.<n>We present the AI Council, a three-phase deliberation framework, and conduct 120 deliberations across two policy scenarios to test two interventions.
  arXiv  Detail & Related papers  (2026-04-29T11:47:28Z)
• A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations [60.2076951536797]
  Large Language Models (LLMs) are increasingly deployed in healthcare scenarios.<n>It is unclear to which extend LLMs could identify and adhere to clinical guidelines during conversations.<n>CPGBench is an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-03-26T09:00:55Z)
• A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Clinician input steers frontier AI models toward both accurate and harmful decisions [10.599240857217811]
  We evaluated 21 large language models (LLMs) across 8 frontier models on differential diagnosis generation and next step recommendations.<n>Expert context significantly improved correct final diagnosis inclusion across all 21 models.<n>In GPT-4o experiments, explicit clinician uncertainty signals improved diagnostic performance after adversarial context.
  arXiv  Detail & Related papers  (2026-03-14T23:47:53Z)
• From Days to Minutes: An Autonomous AI Agent Achieves Reliable Clinical Triage in Remote Patient Monitoring [2.0918370570198763]
  Remote patient monitoring ( RPM) generates vast data, yet landmark trials (Tele-HF, BEAT-HF) failed because data volume overwhelmed clinical staff.<n>We developed Sentinel, an autonomous AI agent using Model Context Protocol (MCP) for contextual triage of RPM vitals.
  arXiv  Detail & Related papers  (2026-03-10T00:50:54Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool [5.585587545595609]
  The purpose of this study was to determine if an ensemble of multiple LLM agents could be used collectively to provide a more reliable assessment of a pixel-based AI triage tool.
  arXiv  Detail & Related papers  (2025-10-30T13:50:19Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)
• MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks [47.486705282473984]
  Large language models (LLMs) achieve near-perfect scores on medical exams.<n>These evaluations inadequately reflect complexity and diversity of real-world clinical practice.<n>We introduce MedHELM, an evaluation framework for assessing LLM performance for medical tasks.
  arXiv  Detail & Related papers  (2025-05-26T22:55:49Z)
• Medical Hallucinations in Foundation Models and Their Impact on Healthcare [71.15392179084428]
  Hallucinations in foundation models arise from autoregressive training objectives.<n>Top-performing models exceeded 97% accuracy when augmented with chain-of-thought prompting.
  arXiv  Detail & Related papers  (2025-02-26T02:30:44Z)
• Closing the gap between open-source and commercial large language models for medical evidence summarization [20.60798771155072]
  Large language models (LLMs) hold great promise in summarizing medical evidence. Most recent studies focus on the application of proprietary LLMs. While open-source LLMs allow better transparency and customization, their performance falls short compared to proprietary ones.
  arXiv  Detail & Related papers  (2024-07-25T05:03:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.