EgoEMS: A High-Fidelity Multimodal Egocentric Dataset for Cognitive Assistance in Emergency Medical Services

• URL: http://arxiv.org/abs/2511.09894v2
• Date: Sat, 15 Nov 2025 14:05:41 GMT
• Title: EgoEMS: A High-Fidelity Multimodal Egocentric Dataset for Cognitive Assistance in Emergency Medical Services
• Authors: Keshara Weerasinghe, Xueren Ge, Tessa Heick, Lahiru Nuwan Wijayasingha, Anthony Cortez, Abhishek Satpathy, John Stankovic, Homa Alemzadeh,
• Abstract summary: EgoEMS is the first end-to-end, high-fidelity, multimodal, multiperson dataset capturing over 20 hours of realistic, procedural EMS activities.<n>Developed in collaboration with EMS experts and aligned with national standards, EgoEMS is captured using an open-source, low-cost, and replicable data collection system.<n>We present a suite of benchmarks for real-time multimodal keystep recognition and action quality estimation, essential for developing AI support tools for EMS.
• Score: 3.0776354206437664
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emergency Medical Services (EMS) are critical to patient survival in emergencies, but first responders often face intense cognitive demands in high-stakes situations. AI cognitive assistants, acting as virtual partners, have the potential to ease this burden by supporting real-time data collection and decision making. In pursuit of this vision, we introduce EgoEMS, the first end-to-end, high-fidelity, multimodal, multiperson dataset capturing over 20 hours of realistic, procedural EMS activities from an egocentric view in 233 simulated emergency scenarios performed by 62 participants, including 46 EMS professionals. Developed in collaboration with EMS experts and aligned with national standards, EgoEMS is captured using an open-source, low-cost, and replicable data collection system and is annotated with keysteps, timestamped audio transcripts with speaker diarization, action quality metrics, and bounding boxes with segmentation masks. Emphasizing realism, the dataset includes responder-patient interactions reflecting real-world emergency dynamics. We also present a suite of benchmarks for real-time multimodal keystep recognition and action quality estimation, essential for developing AI support tools for EMS. We hope EgoEMS inspires the research community to push the boundaries of intelligent EMS systems and ultimately contribute to improved patient outcomes.

Related papers

• LifeEval: A Multimodal Benchmark for Assistive AI in Egocentric Daily Life Tasks [71.05217306468857]
  LifeEval is a multimodal benchmark designed to evaluate real-time, task-oriented human-AI collaboration in daily life.<n>LifeEval emphasizes three key aspects: task-oriented holistic evaluation, egocentric real-time perception from continuous first-person streams, and human-assistant collaborative interaction through natural dialogues.
  arXiv  Detail & Related papers  (2026-02-28T06:05:31Z)
• A Smart-Glasses for Emergency Medical Services via Multimodal Multitask Learning [7.284746127785293]
  We present EMSGlass, a smart-glasses system powered by EMSNet, and EMSServe, a low-latency multimodal serving framework tailored to EMS scenarios.<n>EMSNet integrates text, vital signs, and scene images to construct a unified real-time understanding of EMS incidents.<n>EMSServe achieves 1.9x -- 11.7x speedup over direct PyTorch multimodal inference.
  arXiv  Detail & Related papers  (2025-11-17T07:27:52Z)
• REMONI: An Autonomous System Integrating Wearables and Multimodal Large Language Models for Enhanced Remote Health Monitoring [4.85305127684971]
  This paper proposes REMONI, an autonomous REmote health MONItoring system that integrates multimodal large language models (MLLMs), the Internet of Things (IoT), and wearable devices.<n>The system automatically and continuously collects vital signs, accelerometer data from a special wearable (such as a smartwatch), and visual data in patient video clips collected from cameras.<n>A distinctive feature of our proposed system is the natural language processing component, developed with MLLMs capable of detecting and recognizing a patient's activity and emotion while responding to healthcare worker's inquiries.
  arXiv  Detail & Related papers  (2025-10-24T13:23:38Z)
• Using AI to Optimize Patient Transfer and Resource Utilization During Mass-Casualty Incidents: A Simulation Platform [0.014285185279360277]
  Mass incidents (MCIs) overwhelm healthcare systems and demand rapid patient-hospital allocation decisions.<n>We developed and validated a deep reinforcement learning-based decision-support AI agent to optimize patient transfer decisions.<n>MasTER is a web-accessible command dashboard for MCI management simulations.
  arXiv  Detail & Related papers  (2025-09-10T16:46:54Z)
• EgoBrain: Synergizing Minds and Eyes For Human Action Understanding [50.54007364637855]
  EgoBrain is the world's first large-scale, temporally aligned multimodal dataset that synchronizes egocentric vision and EEG of human brain over extended periods of time.<n>This dataset comprises 61 hours of synchronized 32-channel EEG recordings and first-person video from 40 participants engaged in 29 categories of daily activities.<n>All data, tools, and acquisition protocols are openly shared to foster open science in cognitive computing.
  arXiv  Detail & Related papers  (2025-06-02T06:14:02Z)
• EgoLife: Towards Egocentric Life Assistant [60.51196061794498]
  We introduce EgoLife, a project to develop an egocentric life assistant that accompanies and enhances personal efficiency through AI-powered wearable glasses.<n>We conduct a comprehensive data collection study where six participants lived together for one week, continuously recording their daily activities using AI glasses for multimodal egocentric video capture, along with synchronized third-person-view video references.<n>This effort resulted in the EgoLife dataset, a comprehensive 300-hour egocentric, interpersonal, multiview, and multimodal daily life dataset with intensive annotation.<n>We introduce EgoLifeQA, a suite of long-context, life-oriented question-answering tasks designed to provide
  arXiv  Detail & Related papers  (2025-03-05T18:54:16Z)
• LLMs Can Simulate Standardized Patients via Agent Coevolution [8.539733225671059]
  Training medical personnel using standardized patients (SPs) remains a complex challenge.<n>EvoPatient is a novel simulated patient framework in which a patient agent and doctor agents simulate the diagnostic process through multi-turn dialogues.<n>Our framework improves over existing reasoning methods by more than 10% in requirement alignment and better human preference.
  arXiv  Detail & Related papers  (2024-12-16T12:36:47Z)
• Medchain: Bridging the Gap Between LLM Agents and Clinical Practice with Interactive Sequence [68.05876437208505]
  We present MedChain, a dataset of 12,163 clinical cases that covers five key stages of clinical workflow.<n>We also propose MedChain-Agent, an AI system that integrates a feedback mechanism and a MCase-RAG module to learn from previous cases and adapt its responses.
  arXiv  Detail & Related papers  (2024-12-02T15:25:02Z)
• Real-Time Multimodal Cognitive Assistant for Emergency Medical Services [4.669165383466683]
  This paper presents CognitiveEMS, an end-to-end wearable cognitive assistant system. It can act as a collaborative virtual partner engaging in the real-time acquisition and analysis of multimodal data from an emergency scene.
  arXiv  Detail & Related papers  (2024-03-11T13:56:57Z)
• AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
  We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs. This setup allows for realistic assessments of LLMs in clinical scenarios. We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
  arXiv  Detail & Related papers  (2024-02-15T06:46:48Z)
• Towards Mitigating Hallucination in Large Language Models via Self-Reflection [63.2543947174318]
  Large language models (LLMs) have shown promise for generative and knowledge-intensive tasks including question-answering (QA) tasks. This paper analyses the phenomenon of hallucination in medical generative QA systems using widely adopted LLMs and datasets.
  arXiv  Detail & Related papers  (2023-10-10T03:05:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.