CliniDial: A Naturally Occurring Multimodal Dialogue Dataset for Team Reflection in Action During Clinical Operation

• URL: http://arxiv.org/abs/2506.12936v1
• Date: Sun, 15 Jun 2025 18:39:24 GMT
• Title: CliniDial: A Naturally Occurring Multimodal Dialogue Dataset for Team Reflection in Action During Clinical Operation
• Abstract summary: To understand how the team practices teamwork during the operation, we collected CliniDial from simulations of medical operations.<n>CliniDial includes the audio data and its transcriptions, the simulated physiology signals of the patient manikins, and how the team operates from two camera angles.
• Score: 23.006540048395472
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In clinical operations, teamwork can be the crucial factor that determines the final outcome. Prior studies have shown that sufficient collaboration is the key factor that determines the outcome of an operation. To understand how the team practices teamwork during the operation, we collected CliniDial from simulations of medical operations. CliniDial includes the audio data and its transcriptions, the simulated physiology signals of the patient manikins, and how the team operates from two camera angles. We annotate behavior codes following an existing framework to understand the teamwork process for CliniDial. We pinpoint three main characteristics of our dataset, including its label imbalances, rich and natural interactions, and multiple modalities, and conduct experiments to test existing LLMs' capabilities on handling data with these characteristics. Experimental results show that CliniDial poses significant challenges to the existing models, inviting future effort on developing methods that can deal with real-world clinical data. We open-source the codebase at https://github.com/MichiganNLP/CliniDial

Related papers

• Multi-Stage Patient Role-Playing Framework for Realistic Clinical Interactions [2.1897719729390173]
  We propose the first Chinese patient simulation dataset (Ch-PatientSim)<n>Patients are simulated based on a five-dimensional persona structure.<n>To address issues of the persona class imbalance, a portion of the dataset is augmented using few-shot generation, followed by manual verification.
  arXiv  Detail & Related papers  (2026-01-16T02:34:22Z)
• PRISM2: Unlocking Multi-Modal General Pathology AI with Clinical Dialogue [2.578328028000588]
  We present PRISM2, a multimodal slide-level foundation model trained on data from 700,000 diagnostic specimen-report pairs.<n>PRISM2 aligns histomorphologic features with the language of diagnostic reasoning, producing slide-level representations.<n>Results demonstrate how language-supervised pretraining provides a scalable, clinically grounded signal for learning generalizable pathology representations.
  arXiv  Detail & Related papers  (2025-06-16T03:12:51Z)
• When do they StOP?: A First Step Towards Automatically Identifying Team Communication in the Operating Room [2.508931612863749]
  We generate an OR dataset of real surgeries, called Team-OR, with more than one hundred hours of surgical videos captured by the multi-view camera system in the OR.<n>The dataset contains temporal annotations of 33 Time-out and 22 StOP?-protocol activities in total.<n>We then propose a novel group activity detection approach, where we encode both scene context and action features, and use an efficient neural network model to output the results.
  arXiv  Detail & Related papers  (2025-02-12T10:59:45Z)
• Generalizable Representation Learning for fMRI-based Neurological Disorder Identification [0.0]
  We introduce a novel representation learning strategy to improve the generalization from normal to clinical features.<n>We achieve this by leveraging self-supervised learning on the control dataset to focus on inherent features that are not limited to a particular supervised task.<n>Results demonstrate the superiority of our representation learning strategy on diverse clinically-relevant tasks.
  arXiv  Detail & Related papers  (2024-12-16T22:07:35Z)
• Medchain: Bridging the Gap Between LLM Agents and Clinical Practice through Interactive Sequential Benchmarking [58.25862290294702]
  We present MedChain, a dataset of 12,163 clinical cases that covers five key stages of clinical workflow.<n>We also propose MedChain-Agent, an AI system that integrates a feedback mechanism and a MCase-RAG module to learn from previous cases and adapt its responses.
  arXiv  Detail & Related papers  (2024-12-02T15:25:02Z)
• Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding [53.629132242389716]
  Vision-Language Models (VLM) can support clinicians by analyzing medical images and engaging in natural language interactions. VLMs often exhibit "hallucinogenic" behavior, generating textual outputs not grounded in contextual multimodal information. We propose a new alignment algorithm that uses symbolic representations of clinical reasoning to ground VLMs in medical knowledge.
  arXiv  Detail & Related papers  (2024-05-29T23:19:28Z)
• LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery [57.358568111574314]
  Patient data privacy often restricts the availability of old data when updating the model. Prior CL studies overlooked two vital problems in the surgical domain. This paper proposes addressing these problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology.
  arXiv  Detail & Related papers  (2024-02-26T15:35:24Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Jumpstarting Surgical Computer Vision [2.7396997668655163]
  We employ self-supervised learning to flexibly leverage diverse surgical datasets. We study phase recognition and critical view of safety in laparoscopic cholecystectomy and laparoscopic hysterectomy. The composition of pre-training datasets can severely affect the effectiveness of SSL methods for various downstream tasks.
  arXiv  Detail & Related papers  (2023-12-10T18:54:16Z)
• Development and validation of a natural language processing algorithm to pseudonymize documents in the context of a clinical data warehouse [53.797797404164946]
  The study highlights the difficulties faced in sharing tools and resources in this domain. We annotated a corpus of clinical documents according to 12 types of identifying entities. We build a hybrid system, merging the results of a deep learning model as well as manual rules.
  arXiv  Detail & Related papers  (2023-03-23T17:17:46Z)
• Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning [20.706268332427157]
  An increase in the associated Cognitive Workload (CWL) results from dealing with unexpected and repetitive tasks. In this paper, a cascade of two machine learning approaches is suggested for the multimodal recognition of CWL. A Convolutional Neural Network (CNN) uses this information to identify different types of CWL associated to each surgical task.
  arXiv  Detail & Related papers  (2022-09-12T18:29:34Z)
• Federated Cycling (FedCy): Semi-supervised Federated Learning of Surgical Phases [57.90226879210227]
  FedCy is a semi-supervised learning (FSSL) method that combines FL and self-supervised learning to exploit a decentralized dataset of both labeled and unlabeled videos. We demonstrate significant performance gains over state-of-the-art FSSL methods on the task of automatic recognition of surgical phases.
  arXiv  Detail & Related papers  (2022-03-14T17:44:53Z)
• VBridge: Connecting the Dots Between Features, Explanations, and Data for Healthcare Models [85.4333256782337]
  VBridge is a visual analytics tool that seamlessly incorporates machine learning explanations into clinicians' decision-making workflow. We identified three key challenges, including clinicians' unfamiliarity with ML features, lack of contextual information, and the need for cohort-level evidence. We demonstrated the effectiveness of VBridge through two case studies and expert interviews with four clinicians.
  arXiv  Detail & Related papers  (2021-08-04T17:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.