Longitudinal and Multimodal Recording System to Capture Real-World Patient-Clinician Conversations for AI and Encounter Research: Protocol

• URL: http://arxiv.org/abs/2509.16378v2
• Date: Fri, 26 Sep 2025 16:04:13 GMT
• Title: Longitudinal and Multimodal Recording System to Capture Real-World Patient-Clinician Conversations for AI and Encounter Research: Protocol
• Abstract summary: This study is in an academic outpatient endocrinology clinic at Mayo Clinic.<n>Patients with in-person visits to participating clinicians are invited to enroll. Encounters are recorded with a 360 degree video camera.<n>After each visit, patients complete a survey on empathy, satisfaction, pace, and treatment burden.<n>By August 2025, 35 of 36 eligible clinicians (97%) and 212 of 281 approached patients (75%) had consented. Of encounters, 162 (76%) had complete recordings and 204 (96%) completed the survey.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The promise of AI in medicine depends on learning from data that reflect what matters to patients and clinicians. Most existing models are trained on electronic health records (EHRs), which capture biological measures but rarely patient-clinician interactions. These relationships, central to care, unfold across voice, text, and video, yet remain absent from datasets. As a result, AI systems trained solely on EHRs risk perpetuating a narrow biomedical view of medicine and overlooking the lived exchanges that define clinical encounters. Our objective is to design, implement, and evaluate the feasibility of a longitudinal, multimodal system for capturing patient-clinician encounters, linking 360 degree video/audio recordings with surveys and EHR data to create a dataset for AI research. This single site study is in an academic outpatient endocrinology clinic at Mayo Clinic. Adult patients with in-person visits to participating clinicians are invited to enroll. Encounters are recorded with a 360 degree video camera. After each visit, patients complete a survey on empathy, satisfaction, pace, and treatment burden. Demographic and clinical data are extracted from the EHR. Feasibility is assessed using five endpoints: clinician consent, patient consent, recording success, survey completion, and data linkage across modalities. Recruitment began in January 2025. By August 2025, 35 of 36 eligible clinicians (97%) and 212 of 281 approached patients (75%) had consented. Of consented encounters, 162 (76%) had complete recordings and 204 (96%) completed the survey. This study aims to demonstrate the feasibility of a replicable framework for capturing the multimodal dynamics of patient-clinician encounters. By detailing workflows, endpoints, and ethical safeguards, it provides a template for longitudinal datasets and lays the foundation for AI models that incorporate the complexity of care.

Related papers

• Evidence-Grounded AI for Musculoskeletal Care [106.59121478639402]
  OrthoPilot is a clinical artificial intelligence system powered by a large language model (LLM)<n>It integrates hospital data streams with authoritative external knowledge for continuous musculoskeletal care.<n>In a full-pathway reader study against 81 orthopaedic physicians, OrthoPilot outperformed experts with 25 years of experience.
  arXiv  Detail & Related papers  (2026-07-14T09:03:26Z)
• A multimodal and temporal foundation model for virtual patient representations at healthcare system scale [36.24986458898116]
  We introduce Apollo, a multimodal temporal foundation model trained and evaluated on over three decades of longitudinal hospital records.<n>Apollo learns a unified representation space integrating over 100 thousand unique medical events in our clinical vocabulary.<n>We show that model predictions align with clinically-interpretable multimodal biomarkers.
  arXiv  Detail & Related papers  (2026-04-20T17:55:47Z)
• Beyond Benchmarks of IUGC: Rethinking Requirements of Deep Learning Methods for Intrapartum Ultrasound Biometry from Fetal Ultrasound Videos [58.71502465551297]
  Intrapartum Ultrasound Grand Challenge (IUGC) co-hosted with MICCAI 2024 was launched.<n>IUGC introduces a clinically oriented multi-task automatic measurement framework that integrates standard plane classification, fetal head-pubic symphysis segmentation, and biometry.<n>The challenge releases the largest multi-center intrapartum ultrasound video dataset to date, comprising 774 videos (68,106 frames) collected from three hospitals.
  arXiv  Detail & Related papers  (2026-02-13T13:28:22Z)
• Mitigating Clinician Information Overload: Generative AI for Integrated EHR and RPM Data Analysis [0.523377539745706]
  We present a comprehensive overview of the capabilities, requirements and applications of Generative Artificial Intelligence (GenAI)<n>We first provide some background on the forms and sources of patient data, namely real-time Remote Patient Monitoring ( RPM) streams and traditional Electronic Health Records ( EHRs)<n>These applications can enhance navigation of longitudinal patient data and provide actionable clinical decision support through natural language dialogue.
  arXiv  Detail & Related papers  (2025-08-26T17:10:21Z)
• Barriers in Integrating Medical Visual Question Answering into Radiology Workflows: A Scoping Review and Clinicians' Insights [6.5907034989882725]
  Medical Visual Question Answering (MedVQA) is a promising tool to assist radiologists by automating medical image interpretation through question answering.<n>Despite advances in models and datasets, MedVQA's integration into clinical systems remains limited.<n>This study systematically reviews 68 publications and surveys 50 clinicians from India and Thailand to examine MedVQA's practical utility, challenges, and gaps.
  arXiv  Detail & Related papers  (2025-07-09T09:51:20Z)
• PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions [15.272979678875787]
  We introduce PatientSim, a patient simulator that generates realistic and diverse patient personas for clinical scenarios.<n>PatientSim operates using: 1) clinical profiles, including symptoms and medical history, derived from real-world data in the MIMIC-ED and MIMIC-IV datasets, and 2) personas defined by four axes: personality, language proficiency, medical history recall level, and cognitive confusion level.<n>The top-performing open-source model, Llama 3.3, was validated by four clinicians to confirm the robustness of our framework.
  arXiv  Detail & Related papers  (2025-05-23T12:34:48Z)
• TrialBench: Multi-Modal Artificial Intelligence-Ready Clinical Trial Datasets [54.98321887435557]
  This paper presents a suite of 23 meticulously curated AI-ready datasets covering multi-modal input features and 8 crucial prediction challenges in clinical trial design.<n>We provide basic validation methods for each task to ensure the datasets' usability and reliability.<n>We anticipate that the availability of such open-access datasets will catalyze the development of advanced AI approaches for clinical trial design.
  arXiv  Detail & Related papers  (2024-06-30T09:13:10Z)
• Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding [53.629132242389716]
  Vision-Language Models (VLM) can support clinicians by analyzing medical images and engaging in natural language interactions. VLMs often exhibit "hallucinogenic" behavior, generating textual outputs not grounded in contextual multimodal information. We propose a new alignment algorithm that uses symbolic representations of clinical reasoning to ground VLMs in medical knowledge.
  arXiv  Detail & Related papers  (2024-05-29T23:19:28Z)
• README: Bridging Medical Jargon and Lay Understanding for Patient Education through Data-Centric NLP [9.432205523734707]
  We introduce a new task of automatically generating lay definitions, aiming to simplify medical terms into patient-friendly lay language. We first created the dataset, an extensive collection of over 50,000 unique (medical term, lay definition) pairs and 300,000 mentions. We have also engineered a data-centric Human-AI pipeline that synergizes data filtering, augmentation, and selection to improve data quality.
  arXiv  Detail & Related papers  (2023-12-24T23:01:00Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• LeafAI: query generator for clinical cohort discovery rivaling a human programmer [4.410832512630809]
  We create a system capable of generating data model-agnostic queries. We also provide novel logical reasoning capabilities for complex clinical trial eligibility criteria.
  arXiv  Detail & Related papers  (2023-04-13T00:34:32Z)
• Advancing COVID-19 Diagnosis with Privacy-Preserving Collaboration in Artificial Intelligence [79.038671794961]
  We launch the Unified CT-COVID AI Diagnostic Initiative (UCADI), where the AI model can be distributedly trained and independently executed at each host institution. Our study is based on 9,573 chest computed tomography scans (CTs) from 3,336 patients collected from 23 hospitals located in China and the UK.
  arXiv  Detail & Related papers  (2021-11-18T00:43:41Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)
• COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching [70.08786840301435]
  We propose CrOss-Modal PseudO-SiamEse network (COMPOSE) to address these challenges for patient-trial matching. Experiment results show COMPOSE can reach 98.0% AUC on patient-criteria matching and 83.7% accuracy on patient-trial matching.
  arXiv  Detail & Related papers  (2020-06-15T21:01:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.