MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents

• URL: http://arxiv.org/abs/2607.18999v2
• Date: Sun, 26 Jul 2026 15:35:54 GMT
• Title: MedDDC-Eval: Diagnosis-Decoupled Evaluation of Multi-Turn Medical Consultation Agents
• Abstract summary: We introduce MedDDC-Eval, a diagnosis-decoupled evaluation testbed over held-out cases.<n>It applies the same frozen shared diagnostic reader to every policy-elicited history.<n>It reports diagnostic support, information-acquisition coverage, and efficiency.
• Score: 4.184241883998287
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating multi-turn medical consultation agents requires judging the diagnostic support provided by the histories they elicit through interaction. Yet coupled evaluation lets each policy both elicit the history and generate the terminal diagnosis, so a diagnosis score confounds the elicited history with the policy's own terminal diagnosis generator. We introduce MedDDC-Eval, a diagnosis-decoupled evaluation testbed over held-out cases derived from medical records and online consultations. It applies the same frozen shared diagnostic reader to every policy-elicited history, holding terminal diagnosis generation fixed across policies and enabling comparison under the shared diagnostic reader. It reports diagnostic support, information-acquisition coverage, and efficiency. LLM-assisted semantic matching followed by deterministic one-to-one assignment makes the diagnosis-trajectory-efficiency (D/T/E) scores auditable. In a fixed-history audit across eight policies, replacing each policy's own generator with the shared diagnostic reader shifts diagnosis F1 by 2.2-19.0 points and reverses 18% and 36% of pairwise orderings on the Record and Dialogue splits. To examine downstream utility, we use standard Group Relative Policy Optimization (GRPO) with a separate training-time reward that targets the same diagnosis and trajectory dimensions. Relative to its Qwen3-32B initialization, the trained policy gains 9.6 and 4.6 aggregate-score points on the held-out Record and Dialogue splits, respectively, and ablating either feedback signal reduces the aggregate score on both. Together, MedDDC-Eval supports comparison under a shared diagnostic reader and evaluation-informed policy development, while complementing end-to-end evaluation when terminal diagnosis generation is also part of the target capability.

Related papers

• CDEG: Learning Decision-Critical Evidence for Long-Horizon Diagnostic Agents [21.40725112789917]
  We introduce CDEG, a graph-based framework that learns reusable decision-critical evidence from historical diagnostic trajectories.<n>We show that CDEG consistently improves diagnostic performance, achieving up to an 11.5% accuracy gain over vanilla agents.
  arXiv  Detail & Related papers  (2026-08-24T07:30:22Z)
• A Source-Grounded Framework for Constructing and Evaluating Progressive Multimodal Diagnostic Dialogues from Clinical Case Reports [19.75152371502983]
  We present a source-grounded framework to construct progressive multimodal diagnostic dialogues from case reports.<n> Evaluation on 24 internal medicine case reports showed that our framework can accurately convert case reports into reference dialogues.<n>Results demonstrate that fluent responses do not necessarily reflect evidence-grounded clinical reasoning.
  arXiv  Detail & Related papers  (2026-08-24T02:04:39Z)
• Beyond the Leaderboard: Design Lessons for Trustworthy Multimodal VQA [4.080704298118247]
  multimodal AI must combine visual and textual evidence while remaining reliable and interpretable.<n>We analyze design choices across nine documented systems for question answering and explanation quality.<n>The findings support trustworthy multimodal healthcare AI based on data fusion, explainability, and resilient evaluation.
  arXiv  Detail & Related papers  (2026-07-16T17:38:19Z)
• PACT: Learning Diverse Diagnostic Strategies via Privileged Synthesis and Branch Consensus [31.605826007992167]
  We propose textbfPACT (Periodic Anchor Consensus Training), a framework that couples supervised multi-paradigm dialogue synthesis with consensus-based Branch training.<n>At the data level, textbfDPS (Doctor-Patient-Supervisor) uses complete electronic medical records (EMRs) for quality control while keeping the doctor agent restricted to patient-visible information.<n>Experiments show that PACT achieves state-of-the-art performance among compared proprietary, medical-specialized, and task-adapted baselines on diagnostic outcome and consultation-process metrics.
  arXiv  Detail & Related papers  (2026-06-08T02:34:12Z)
• Clinically-Grounded Counterfactual Reasoning for Medical Video Diagnosis [47.559337485629904]
  This work introduces MedVCR, a counterfactual reasoning framework that mimics clinical diagnostic thinking.<n> MedVCR comprises three components: a Counterfactual Generator that synthesizes tissue evolution under specified pathological states.<n>It is evaluated under both fully supervised (e.g., colposcopy) and weakly supervised (e.g., colonoscopy) video diagnosis settings, yielding 2.6%-10.2% performance gains.
  arXiv  Detail & Related papers  (2026-05-26T02:46:15Z)
• Benchmarking Multi-turn Medical Diagnosis: Hold, Lure, and Self-Correction [72.89352076103889]
  Large language models (LLMs) achieve high accuracy in medical diagnosis when all clinical information is provided in a single turn.<n>We introduce MINT, a high-fidelity, multi-turn medical diagnosis benchmark comprising 1,035 cases.<n>We uncover three persistent behavioral patterns that significantly impact diagnostic decisions.
  arXiv  Detail & Related papers  (2026-04-06T00:23:10Z)
• Optimizing Point-of-Care Ultrasound Video Acquisition for Probabilistic Multi-Task Heart Failure Detection [3.5617951813421818]
  We introduce a personalized data acquisition strategy in which an RL agent selects the next view to acquire or terminates acquisition.<n>Upon termination, a diagnostic model jointly predicts aortic stenosis (AS) severity and left ventricular ejection fraction (LVEF)<n>Our method matches full-study performance while using 32% fewer videos, achieving 77.2% mean balanced accuracy (bACC) across AS severity classification and LVEF estimation.
  arXiv  Detail & Related papers  (2026-02-14T07:56:21Z)
• LingxiDiagBench: A Multi-Agent Framework for Benchmarking LLMs in Chinese Psychiatric Consultation and Diagnosis [14.82377002030236]
  Mental disorders are highly prevalent worldwide.<n>The shortage of psychiatrists and the inherent subjectivity of interview-based diagnosis create substantial barriers to timely and consistent mental-health assessment.<n>We present LingxiDiagBench, a large-scale multi-agent benchmark.
  arXiv  Detail & Related papers  (2026-02-10T03:46:05Z)
• MedAD-R1: Eliciting Consistent Reasoning in Interpretible Medical Anomaly Detection via Consistency-Reinforced Policy Optimization [46.65200216642429]
  We introduce MedAD-38K, the first large-scale, multi-modal, and multi-center benchmark for MedAD featuring diagnostic Chain-of-Thought (CoT) annotations alongside structured Visual Question-Answering (VQA) pairs.<n>Our proposed model, MedAD-R1, achieves state-of-the-art (SOTA) performance on the MedAD-38K benchmark, outperforming strong baselines by more than 10%.
  arXiv  Detail & Related papers  (2026-02-01T07:56:10Z)
• Towards Reliable Medical LLMs: Benchmarking and Enhancing Confidence Estimation of Large Language Models in Medical Consultation [97.36081721024728]
  We propose the first benchmark for assessing confidence in multi-turn interaction during realistic medical consultations.<n>Our benchmark unifies three types of medical data for open-ended diagnostic generation.<n>We present MedConf, an evidence-grounded linguistic self-assessment framework.
  arXiv  Detail & Related papers  (2026-01-22T04:51:39Z)
• MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosis [13.241795322837861]
  We introduce MedEinst, a counterfactual benchmark with 5,383 paired clinical cases across 49 diseases.<n>We measure susceptibility via Bias Trap Rate--probability of misdiagnosing traps despite correctly diagnosing controls.
  arXiv  Detail & Related papers  (2026-01-10T17:39:25Z)
• Evolving Diagnostic Agents in a Virtual Clinical Environment [75.59389103511559]
  We present a framework for training large language models (LLMs) as diagnostic agents with reinforcement learning.<n>Our method acquires diagnostic strategies through interactive exploration and outcome-based feedback.<n>DiagAgent significantly outperforms 10 state-of-the-art LLMs, including DeepSeek-v3 and GPT-4o.
  arXiv  Detail & Related papers  (2025-10-28T17:19:47Z)
• End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoning [52.12425911708585]
  Deep-DxSearch is an agentic RAG system trained end-to-end with reinforcement learning (RL)<n>In Deep-DxSearch, we first construct a large-scale medical retrieval corpus comprising patient records and reliable medical knowledge sources.<n> Experiments demonstrate that our end-to-end RL training framework consistently outperforms prompt-engineering and training-free RAG approaches.
  arXiv  Detail & Related papers  (2025-08-21T17:42:47Z)
• MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot [47.77948063906033]
  Retrieval-augmented generation (RAG) is a well-suited technique for retrieving privacy-sensitive Electronic Health Records.<n>This paper proposes MedRAG, a RAG model enhanced by knowledge graph (KG)-elicited reasoning for the medical domain.<n>Tests show MedRAG provides more specific diagnostic insights and outperforms state-of-the-art models in reducing misdiagnosis rates.
  arXiv  Detail & Related papers  (2025-02-06T12:27:35Z)
• Extracting Diagnosis Pathways from Electronic Health Records Using Deep Reinforcement Learning [2.0191844627740254]
  We aim to learn the optimal sequence of actions to perform in order to obtain a correct diagnosis from electronic health records. We apply various deep reinforcement learning algorithms to this task and experiment on a synthetic but realistic dataset to differentially diagnose anemia.
  arXiv  Detail & Related papers  (2023-05-10T16:36:54Z)
• Semi-Supervised Variational Reasoning for Medical Dialogue Generation [70.838542865384]
  Two key characteristics are relevant for medical dialogue generation: patient states and physician actions. We propose an end-to-end variational reasoning approach to medical dialogue generation. A physician policy network composed of an action-classifier and two reasoning detectors is proposed for augmented reasoning ability.
  arXiv  Detail & Related papers  (2021-05-13T04:14:35Z)
• Towards Causality-Aware Inferring: A Sequential Discriminative Approach for Medical Diagnosis [142.90770786804507]
  Medical diagnosis assistant (MDA) aims to build an interactive diagnostic agent to sequentially inquire about symptoms for discriminating diseases. This work attempts to address these critical issues in MDA by taking advantage of the causal diagram. We propose a propensity-based patient simulator to effectively answer unrecorded inquiry by drawing knowledge from the other records.
  arXiv  Detail & Related papers  (2020-03-14T02:05:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.