MATRIX: Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation

• URL: http://arxiv.org/abs/2508.19163v1
• Date: Tue, 26 Aug 2025 16:12:12 GMT
• Title: MATRIX: Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation
• Abstract summary: MATRIX is a structured engineering framework for safety-oriented evaluation of clinical dialogue agents.<n>It integrates a safety-aligned taxonomy of clinical scenarios, expected system behaviors and failure modes, BehvJudge, an evaluator for detecting safety-relevant dialogue failures, and PatBot, a simulated patient agent.<n>Across three experiments, we show that MATRIX enables systematic, scalable safety evaluation.
• Score: 3.9146063017280923
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite the growing use of large language models (LLMs) in clinical dialogue systems, existing evaluations focus on task completion or fluency, offering little insight into the behavioral and risk management requirements essential for safety-critical systems. This paper presents MATRIX (Multi-Agent simulaTion fRamework for safe Interactions and conteXtual clinical conversational evaluation), a structured, extensible framework for safety-oriented evaluation of clinical dialogue agents. MATRIX integrates three components: (1) a safety-aligned taxonomy of clinical scenarios, expected system behaviors and failure modes derived through structured safety engineering methods; (2) BehvJudge, an LLM-based evaluator for detecting safety-relevant dialogue failures, validated against expert clinician annotations; and (3) PatBot, a simulated patient agent capable of producing diverse, scenario-conditioned responses, evaluated for realism and behavioral fidelity with human factors expertise, and a patient-preference study. Across three experiments, we show that MATRIX enables systematic, scalable safety evaluation. BehvJudge with Gemini 2.5-Pro achieves expert-level hazard detection (F1 0.96, sensitivity 0.999), outperforming clinicians in a blinded assessment of 240 dialogues. We also conducted one of the first realism analyses of LLM-based patient simulation, showing that PatBot reliably simulates realistic patient behavior in quantitative and qualitative evaluations. Using MATRIX, we demonstrate its effectiveness in benchmarking five LLM agents across 2,100 simulated dialogues spanning 14 hazard scenarios and 10 clinical domains. MATRIX is the first framework to unify structured safety engineering with scalable, validated conversational AI evaluation, enabling regulator-aligned safety auditing. We release all evaluation tools, prompts, structured scenarios, and datasets.

Related papers

• Evaluating Patient Safety Risks in Generative AI: Development and Validation of a FMECA Framework for Generated Clinical Content [0.5049603364708264]
  Failure Mode, Effects, and Criticality Analysis (FMECA) provides a proactive framework for systematic risk identification.<n>This study aimed to develop and validate a novel FMECA framework for the prospective assessment of patient safety risks in LLM-generated clinical summaries.
  arXiv  Detail & Related papers  (2026-04-23T14:44:15Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming [23.573537738272595]
  We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with cognitive-affective models.<n>We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents.<n>Our large-scale simulation reveals critical safety gaps in the use of AI for mental health support.
  arXiv  Detail & Related papers  (2026-02-23T15:17:18Z)
• AgentsEval: Clinically Faithful Evaluation of Medical Imaging Reports via Multi-Agent Reasoning [73.50200033931148]
  We introduce AgentsEval, a multi-agent stream reasoning framework that emulates the collaborative diagnostic workflow of radiologists.<n>By dividing the evaluation process into interpretable steps including criteria definition, evidence extraction, alignment, and consistency scoring, AgentsEval provides explicit reasoning traces and structured clinical feedback.<n> Experimental results demonstrate that AgentsEval delivers clinically aligned, semantically faithful, and interpretable evaluations that remain robust under paraphrastic, semantic, and stylistic perturbations.
  arXiv  Detail & Related papers  (2026-01-23T11:59:13Z)
• Bridging the Knowledge-Action Gap by Evaluating LLMs in Dynamic Dental Clinical Scenarios [9.865786198063644]
  The transition of Large Language Models (LLMs) from passive knowledge retrievers to autonomous clinical agents demands a shift in evaluation-from static accuracy to dynamic behavioral reliability.<n>This study empirically charts the capability boundaries of dental LLMs, providing a roadmap for bridging the gap between standardized knowledge and safe, autonomous clinical practice.
  arXiv  Detail & Related papers  (2026-01-19T11:36:39Z)
• Benchmarking Egocentric Clinical Intent Understanding Capability for Medical Multimodal Large Language Models [48.95516224614331]
  We introduce MedGaze-Bench, the first benchmark leveraging clinician gaze as a Cognitive Cursor to assess intent understanding across surgery, emergency simulation, and diagnostic interpretation.<n>Our benchmark addresses three fundamental challenges: visual homogeneity of anatomical structures, strict temporal-causal dependencies in clinical, and implicit adherence to safety protocols.
  arXiv  Detail & Related papers  (2026-01-11T02:20:40Z)
• ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning [58.01333341218153]
  We propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues.<n>Our method generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent.<n>Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-12-29T12:58:58Z)
• AutoMedic: An Automated Evaluation Framework for Clinical Conversational Agents with Medical Dataset Grounding [4.87216588304398]
  We introduce AutoMedic, a multi-agent simulation framework that enables automated evaluation of large language models (LLMs) as clinical conversational agents.<n>AutoMedic transforms off-the-shelf static QA datasets into virtual patient profiles, enabling realistic and clinically grounded multi-turn clinical dialogues.<n>The performance of various clinical conversational agents is then assessed based on our CARE metric, which provides a multi-faceted evaluation standard of clinical conversational accuracy, efficiency/strategy, empathy, and robustness.
  arXiv  Detail & Related papers  (2025-12-11T01:25:36Z)
• WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue [3.468314243424983]
  Automatic Speech Recognition (ASR) is increasingly deployed in clinical dialogue.<n>Standard evaluations still rely heavily on Error Error Rate (WER)<n>This paper challenges that standard, investigating whether WER or other common metrics correlate with clinical impact of transcription errors.
  arXiv  Detail & Related papers  (2025-11-20T16:59:20Z)
• Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Models [51.91760712805404]
  We introduce VivaBench, a benchmark for evaluating sequential clinical reasoning in large language models (LLMs)<n>Our dataset consists of 1762 physician-curated clinical vignettes structured as interactive scenarios that simulate a (oral) examination in medical training.<n>Our analysis identified several failure modes that mirror common cognitive errors in clinical practice.
  arXiv  Detail & Related papers  (2025-10-11T16:24:35Z)
• A Risk Taxonomy for Evaluating AI-Powered Psychotherapy Agents [10.405048273969085]
  We introduce a novel risk taxonomy specifically designed for the systematic evaluation of conversational AI psychotherapists.<n>We discuss two use cases in detail: monitoring cognitive model-based risk factors during a counseling conversation to detect unsafe deviations, and in automated benchmarking of AI psychotherapists with simulated patients.
  arXiv  Detail & Related papers  (2025-05-21T05:01:39Z)
• Ψ-Arena: Interactive Assessment and Optimization of LLM-based Psychological Counselors with Tripartite Feedback [51.26493826461026]
  We propose Psi-Arena, an interactive framework for comprehensive assessment and optimization of large language models (LLMs)<n>Arena features realistic arena interactions that simulate real-world counseling through multi-stage dialogues with psychologically profiled NPC clients.<n>Experiments across eight state-of-the-art LLMs show significant performance variations in different real-world scenarios and evaluation perspectives.
  arXiv  Detail & Related papers  (2025-05-06T08:22:51Z)
• Med-CoDE: Medical Critique based Disagreement Evaluation Framework [72.42301910238861]
  The reliability and accuracy of large language models (LLMs) in medical contexts remain critical concerns.<n>Current evaluation methods often lack robustness and fail to provide a comprehensive assessment of LLM performance.<n>We propose Med-CoDE, a specifically designed evaluation framework for medical LLMs to address these challenges.
  arXiv  Detail & Related papers  (2025-04-21T16:51:11Z)
• Self-Evolving Multi-Agent Simulations for Realistic Clinical Interactions [16.50490537786593]
  We introduce MedAgentSim, an open-source simulated clinical environment with doctor, patient, and measurement agents.<n>Unlike prior approaches, our framework requires doctor agents to actively engage with patients through multi-turn conversations.<n>We incorporate self improvement mechanisms that allow models to iteratively refine their diagnostic strategies.
  arXiv  Detail & Related papers  (2025-03-28T17:59:53Z)
• Quantifying the Reasoning Abilities of LLMs on Real-world Clinical Cases [48.87360916431396]
  We introduce MedR-Bench, a benchmarking dataset of 1,453 structured patient cases, annotated with reasoning references.<n>We propose a framework encompassing three critical examination recommendation, diagnostic decision-making, and treatment planning, simulating the entire patient care journey.<n>Using this benchmark, we evaluate five state-of-the-art reasoning LLMs, including DeepSeek-R1, OpenAI-o3-mini, and Gemini-2.0-Flash Thinking, etc.
  arXiv  Detail & Related papers  (2025-03-06T18:35:39Z)
• ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems [0.0]
  Large Language Models (LLMs) have shown impressive potential in clinical question answering.<n>RAG is emerging as a leading approach for ensuring the factual accuracy of model responses.<n>Current automated RAG metrics perform poorly in clinical and conversational use cases.
  arXiv  Detail & Related papers  (2025-01-14T15:46:39Z)
• Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm [15.627870862369784]
  Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis. We propose an automatic evaluation paradigm tailored to assess the LLMs' capabilities in delivering clinical services.
  arXiv  Detail & Related papers  (2024-03-25T06:17:54Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.