Yes, this is what I was looking for! Towards Multi-modal Medical
Consultation Concern Summary Generation
- URL: http://arxiv.org/abs/2401.05134v1
- Date: Wed, 10 Jan 2024 12:56:47 GMT
- Title: Yes, this is what I was looking for! Towards Multi-modal Medical
Consultation Concern Summary Generation
- Authors: Abhisek Tiwari, Shreyangshu Bera, Sriparna Saha, Pushpak
Bhattacharyya, Samrat Ghosh
- Abstract summary: We propose a new task of multi-modal medical concern summary generation.
Nonverbal cues, such as patients' gestures and facial expressions, aid in accurately identifying patients' concerns.
We construct the first multi-modal medical concern summary generation corpus.
- Score: 46.42604861624895
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Over the past few years, the use of the Internet for healthcare-related tasks
has grown by leaps and bounds, posing a challenge in effectively managing and
processing information to ensure its efficient utilization. During moments of
emotional turmoil and psychological challenges, we frequently turn to the
internet as our initial source of support, choosing this over discussing our
feelings with others due to the associated social stigma. In this paper, we
propose a new task of multi-modal medical concern summary (MMCS) generation,
which provides a short and precise summary of patients' major concerns brought
up during the consultation. Nonverbal cues, such as patients' gestures and
facial expressions, aid in accurately identifying patients' concerns. Doctors
also consider patients' personal information, such as age and gender, in order
to describe the medical condition appropriately. Motivated by the potential
efficacy of patients' personal context and visual gestures, we propose a
transformer-based multi-task, multi-modal intent-recognition, and medical
concern summary generation (IR-MMCSG) system. Furthermore, we propose a
multitasking framework for intent recognition and medical concern summary
generation for doctor-patient consultations. We construct the first multi-modal
medical concern summary generation (MM-MediConSummation) corpus, which includes
patient-doctor consultations annotated with medical concern summaries, intents,
patient personal information, doctor's recommendations, and keywords. Our
experiments and analysis demonstrate (a) the significant role of patients'
expressions/gestures and their personal information in intent identification
and medical concern summary generation, and (b) the strong correlation between
intent recognition and patients' medical concern summary generation
The dataset and source code are available at https://github.com/NLP-RL/MMCSG.
Related papers
- MedInsight: A Multi-Source Context Augmentation Framework for Generating
Patient-Centric Medical Responses using Large Language Models [3.0874677990361246]
Large Language Models (LLMs) have shown impressive capabilities in generating human-like responses.
We propose MedInsight:a novel retrieval framework that augments LLM inputs with relevant background information.
Experiments on the MTSamples dataset validate MedInsight's effectiveness in generating contextually appropriate medical responses.
arXiv Detail & Related papers (2024-03-13T15:20:30Z) - Are Generative AI systems Capable of Supporting Information Needs of
Patients? [4.485098382568721]
We investigate whether and how generative visual question answering systems can responsibly support patient information needs in the context of radiology imaging data.
We conducted a formative need-finding study in which participants discussed chest computed tomography (CT) scans and associated radiology reports of a fictitious close relative with a cardiothoracic radiologist.
Using thematic analysis of the conversation between participants and medical experts, we identified commonly occurring themes across interactions.
We evaluate two state-of-the-art generative visual language models against the radiologist's responses.
arXiv Detail & Related papers (2024-01-31T23:24:37Z) - MedSumm: A Multimodal Approach to Summarizing Code-Mixed Hindi-English
Clinical Queries [16.101969130235055]
We introduce the Multimodal Medical Codemixed Question Summarization MMCQS dataset.
This dataset combines Hindi-English codemixed medical queries with visual aids.
Our dataset, code, and pre-trained models will be made publicly available.
arXiv Detail & Related papers (2024-01-03T07:58:25Z) - Experience and Evidence are the eyes of an excellent summarizer! Towards
Knowledge Infused Multi-modal Clinical Conversation Summarization [46.613541673040544]
We propose a knowledge-infused, multi-modal, multi-tasking medical domain identification and clinical conversation summary generation framework.
We develop a multi-modal, multi-intent clinical conversation summarization corpus annotated with intent, symptom, and summary.
The extensive set of experiments led to the following findings: (a) critical significance of visuals, (b) more precise and medical entity preserving summary with additional knowledge infusion, and (c) a correlation between medical department identification and clinical synopsis generation.
arXiv Detail & Related papers (2023-09-27T15:49:43Z) - MD-Manifold: A Medical-Distance-Based Representation Learning Approach
for Medical Concept and Patient Representation [6.795388490479779]
Representing medical concepts for healthcare analytical tasks requires incorporating medical domain knowledge and prior data information.
Our proposed framework, MD-Manifold, introduces a novel approach to medical concept and patient representation.
It includes a new data augmentation approach, concept distance metric, and patient-patient network to incorporate crucial medical domain knowledge and prior data information.
arXiv Detail & Related papers (2023-04-30T18:58:32Z) - Towards Medical Artificial General Intelligence via Knowledge-Enhanced
Multimodal Pretraining [121.89793208683625]
Medical artificial general intelligence (MAGI) enables one foundation model to solve different medical tasks.
We propose a new paradigm called Medical-knedge-enhanced mulTimOdal pretRaining (MOTOR)
arXiv Detail & Related papers (2023-04-26T01:26:19Z) - A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks
and Datasets [70.32630628211803]
We propose two frameworks to support automatic medical consultation, namely doctor-patient dialogue understanding and task-oriented interaction.
A new large medical dialogue dataset with multi-level fine-grained annotations is introduced.
We report a set of benchmark results for each task, which shows the usability of the dataset and sets a baseline for future studies.
arXiv Detail & Related papers (2022-04-19T16:43:21Z) - MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware
Medical Dialogue Generation [86.38736781043109]
We build and release a large-scale high-quality Medical Dialogue dataset related to 12 types of common Gastrointestinal diseases named MedDG.
We propose two kinds of medical dialogue tasks based on MedDG dataset. One is the next entity prediction and the other is the doctor response generation.
Experimental results show that the pre-train language models and other baselines struggle on both tasks with poor performance in our dataset.
arXiv Detail & Related papers (2020-10-15T03:34:33Z) - BiteNet: Bidirectional Temporal Encoder Network to Predict Medical
Outcomes [53.163089893876645]
We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey.
An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys.
We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
arXiv Detail & Related papers (2020-09-24T00:42:36Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.