An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography

• URL: http://arxiv.org/abs/2608.07651v1
• Date: Fri, 07 Aug 2026 17:33:12 GMT
• Title: An Agentic AI Framework Overcomes Fundamental Limitations of Large Language Models for Glaucoma Detection from Fundus Photography
• Abstract summary: Large language models (LLMs) show promise in medical image interpretation but suffer from hallucination, limited accuracy, and run-to-run inconsistency.<n>We developed and validated an agentic AI framework integrating LLMs with specialized deep learning tools for glaucoma detection from fundus photography.
• Score: 1.3023698730561286
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) show promise in medical image interpretation but suffer from hallucination, limited accuracy, and run-to-run inconsistency. We developed and validated an agentic AI framework integrating LLMs with specialized deep learning tools for glaucoma detection from fundus photography. The workflow had three steps: (1) LLM initial assessment; (2) function calling to invoke specialized tools for image quality (QAModel, FundaQ-8), glaucoma classification (SwinV2-Tiny), and optic disc/cup segmentation (SegFormer-B0); and (3) LLM reflection integrating the initial impression with tool outputs. Two LLMs (Gemini 2.5 Flash, GPT-5.4 mini) were evaluated on two public datasets (ORIGA, n=100; RIM-ONE-v3, n=100) under uncropped and cropped fields of view; all images were independently graded by a masked fellowship-trained glaucoma specialist. The agentic workflow improved classification accuracy by 16 to 47 percentage points across all conditions, reaching within 6 points of the specialist; on RIM-ONE-v3 the best configurations matched the specialist accuracy of 88%. LLM-alone approaches failed in two ways: GPT-5.4 mini showed positive bias (sensitivity 95-100%, specificity 0-5%), while Gemini 2.5 Flash varied stochastically between runs; the agentic workflow corrected both. Cup-to-disc ratio error fell 15-50% (MAE 0.156-0.228 to 0.104-0.132), and correlation with specialist grading rose from weak (r=0.12-0.39) to moderate-strong (r=0.59-0.84). Run-to-run consistency rose from near-random (kappa as low as -0.01) to near-perfect (kappa up to 0.96). Integrating LLMs with specialized tools addressed key limitations of LLM-alone approaches, including over-diagnosis and run-to-run variability. Gains held for both LLMs, suggesting generalizability across backbones, and may signal a shift from monolithic models toward orchestrated multi-agent systems in medical AI.

Related papers

• Fusing Perceptual Vision Experts with Multimodal Large Language Models for Explainable Plant Disease Diagnosis: From Benchmark Imagery to Real-World Robotic Field Validation [3.7105561711383266]
  We present the Hybrid Hierarchical Multi-Agent Framework (H$2$MAF), combining decision-level fusion of EfficientNet-B3 and ConvNeXt-Tiny with semantic arbitration.<n>On PlantDoc, Gemma improves accuracy from 63.9% to 68.5%, achieving +7.6 points on the 41.7% CNN-conflict subset.<n>These results establish MLLM arbitration as a promising, yet calibration-dependent, approach for explainable agricultural AI and robotic field decision support.
  arXiv  Detail & Related papers  (2026-08-23T17:20:20Z)
• Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge [46.547776251310694]
  This paper evaluates 5G domain understanding and fault analysis in a free-text generation format.<n>We evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks.
  arXiv  Detail & Related papers  (2026-08-21T12:09:51Z)
• ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral [3.306924107222105]
  We investigate whether a generalist large language model (LLM), reading only a faithful natural-language rendering of standard nodule attributes, can serve as a calibrated triage layer.<n>We propose ConfTriage, a confidence-calibrated method built on three pillars: language as the modality, calibration as the safety mechanism, and a selective specialist DL backstop for low-confidence cases.
  arXiv  Detail & Related papers  (2026-08-11T13:02:41Z)
• Perception, Verdict, and Evolution: Hindsight-Driven Self-Refining Forensics Agent for AI-Generated Image Detection [80.15317858033534]
  ForeAgent is an agentic forensics framework for AI-generated image detection with iterative self-evolution.<n>ForeAgent achieves state-of-the-art performance on the Chameleon benchmark, reaching 82.18% accuracy.<n>ForeAgent produces more consistent and causally grounded reasoning compared to GPT-5 and GPT-5-mini.
  arXiv  Detail & Related papers  (2026-06-25T02:59:33Z)
• LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data [2.1443570696048906]
  Large language models (LLMs) are increasingly applied to structured clinical data.<n>We compare Qwen 2.5 7B and XGBoost on a prediction task via attribution divergence analysis.
  arXiv  Detail & Related papers  (2026-06-17T18:49:44Z)
• Vision-Language Models as Zero-Annotation Oracles in Histopathology [10.853300035340615]
  Foreground segmentation is the critical first step of every computational pathology pipeline.<n>We propose a coarse-to-fine approach that recasts foreground segmentation as a visual perception task.<n>Our framework provides a principled, scalable solution to a persistent infrastructure bottleneck in digital pathology.
  arXiv  Detail & Related papers  (2026-06-15T12:50:19Z)
• A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks [46.43161388634735]
  Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for years.<n>We audit open VLMs on SLAKE-En, PathVQA, VQA-RAD, and an auxiliary public OmniMedVQA mirror using four detector families.
  arXiv  Detail & Related papers  (2026-06-08T18:40:10Z)
• A Multi-agent Large Language Model Framework to Automatically Assess Performance of a Clinical AI Triage Tool [5.585587545595609]
  The purpose of this study was to determine if an ensemble of multiple LLM agents could be used collectively to provide a more reliable assessment of a pixel-based AI triage tool.
  arXiv  Detail & Related papers  (2025-10-30T13:50:19Z)
• MDF-MLLM: Deep Fusion Through Cross-Modal Feature Alignment for Contextually Aware Fundoscopic Image Classification [0.32622301272834514]
  Existing multimodal large language models (MLLMs) often struggle to capture low-level spatial details critical for diagnosing retinal diseases.<n>This model development and validation study was conducted on 1,305 fundus image-text pairs compiled from three public datasets.<n> MDF-MLLM integrates skip features from four U-Net layers encoder into cross-attention blocks within a LLaMA 3.2 11B MLLM.
  arXiv  Detail & Related papers  (2025-09-21T05:46:35Z)
• Enhancement Without Contrast: Stability-Aware Multicenter Machine Learning for Glioma MRI Imaging [2.5012408467295555]
  Predicting contrast enhancement from non-contrast MRI using machine learning (ML) offers a safer alternative to Gadolinium-based contrast agents (GBCAs)<n>We propose a stability-aware framework to identify reproducible ML pipelines for multicenter prediction of glioma MRI contrast enhancement.
  arXiv  Detail & Related papers  (2025-09-13T00:47:07Z)
• LLM2: Let Large Language Models Harness System 2 Reasoning [65.89293674479907]
  Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs.<n>We introduce LLM2, a novel framework that combines an LLM with a process-based verifier.<n>LLMs2 is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs.
  arXiv  Detail & Related papers  (2024-12-29T06:32:36Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• LLM Robustness Against Misinformation in Biomedical Question Answering [50.98256373698759]
  The retrieval-augmented generation (RAG) approach is used to reduce the confabulation of large language models (LLMs) for question answering. We evaluate the effectiveness and robustness of four LLMs against misinformation in answering biomedical questions.
  arXiv  Detail & Related papers  (2024-10-27T16:23:26Z)
• RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness [88.82621231987815]
  We introduce RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm.<n> RLAIF-V maximally explores open-source MLLMs from two perspectives, including high-quality feedback data generation.<n>Experiments on six benchmarks in both automatic and human evaluation show that RLAIF-V substantially enhances the trustworthiness of models.
  arXiv  Detail & Related papers  (2024-05-27T14:37:01Z)
• "Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation [90.09260023184932]
  Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. NoMIRACL is a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. We measure relevance assessment using: (i) hallucination rate, measuring model tendency to hallucinate, when the answer is not present in passages in the non-relevant subset, and (ii) error rate, measuring model inaccuracy to recognize relevant passages in the relevant subset.
  arXiv  Detail & Related papers  (2023-12-18T17:18:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.