Overview of TREC 2024 Medical Video Question Answering (MedVidQA) Track

• URL: http://arxiv.org/abs/2412.11056v1
• Date: Sun, 15 Dec 2024 05:18:01 GMT
• Title: Overview of TREC 2024 Medical Video Question Answering (MedVidQA) Track
• Authors: Deepak Gupta, Dina Demner-Fushman,
• Abstract summary: We introduce new tasks to foster research toward designing systems that can understand medical videos to provide visual answers to natural language questions. These tasks have the potential to support the development of sophisticated downstream applications that can benefit the public and medical professionals.
• Score: 19.06634510937997
• License:
• Abstract: One of the key goals of artificial intelligence (AI) is the development of a multimodal system that facilitates communication with the visual world (image and video) using a natural language query. Earlier works on medical question answering primarily focused on textual and visual (image) modalities, which may be inefficient in answering questions requiring demonstration. In recent years, significant progress has been achieved due to the introduction of large-scale language-vision datasets and the development of efficient deep neural techniques that bridge the gap between language and visual understanding. Improvements have been made in numerous vision-and-language tasks, such as visual captioning visual question answering, and natural language video localization. Most of the existing work on language vision focused on creating datasets and developing solutions for open-domain applications. We believe medical videos may provide the best possible answers to many first aid, medical emergency, and medical education questions. With increasing interest in AI to support clinical decision-making and improve patient engagement, there is a need to explore such challenges and develop efficient algorithms for medical language-video understanding and generation. Toward this, we introduced new tasks to foster research toward designing systems that can understand medical videos to provide visual answers to natural language questions, and are equipped with multimodal capability to generate instruction steps from the medical video. These tasks have the potential to support the development of sophisticated downstream applications that can benefit the public and medical professionals.

Related papers

• A Survey of Medical Vision-and-Language Applications and Their Techniques [48.268198631277315]
  Medical vision-and-language models (MVLMs) have attracted substantial interest due to their capability to offer a natural language interface for interpreting complex medical data. Here, we provide a comprehensive overview of MVLMs and the various medical tasks to which they have been applied. We also examine the datasets used for these tasks and compare the performance of different models based on standardized evaluation metrics.
  arXiv  Detail & Related papers  (2024-11-19T03:27:05Z)
• Developing ChatGPT for Biology and Medicine: A Complete Review of Biomedical Question Answering [25.569980942498347]
  ChatGPT explores a strategic blueprint of question answering (QA) in delivering medical diagnosis, treatment recommendations, and other healthcare support. This is achieved through the increasing incorporation of medical domain data via natural language processing (NLP) and multimodal paradigms.
  arXiv  Detail & Related papers  (2024-01-15T07:21:16Z)
• Towards Answering Health-related Questions from Medical Videos: Datasets and Approaches [21.16331827504689]
  A growing number of individuals now prefer instructional videos as they offer a series of step-by-step procedures to accomplish particular tasks. The instructional videos from the medical domain may provide the best possible visual answers to first aid, medical emergency, and medical education questions. The scarcity of large-scale datasets in the medical domain is a key challenge that hinders the development of applications that can help the public with their health-related questions.
  arXiv  Detail & Related papers  (2023-09-21T16:21:28Z)
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures [51.78027546947034]
  Recent advancements in surgical computer vision have been driven by vision-only models, which lack language semantics. We propose leveraging surgical video lectures from e-learning platforms to provide effective vision and language supervisory signals. We address surgery-specific linguistic challenges using multiple automatic speech recognition systems for text transcriptions.
  arXiv  Detail & Related papers  (2023-07-27T22:38:12Z)
• XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models [60.437091462613544]
  We introduce XrayGPT, a novel conversational medical vision-language model. It can analyze and answer open-ended questions about chest radiographs. We generate 217k interactive and high-quality summaries from free-text radiology reports.
  arXiv  Detail & Related papers  (2023-06-13T17:59:59Z)
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day [85.19963303642427]
  We propose a cost-efficient approach for training a vision-language conversational assistant that can answer open-ended research questions of biomedical images. The model first learns to align biomedical vocabulary using the figure-caption pairs as is, then learns to master open-ended conversational semantics. This enables us to train a Large Language and Vision Assistant for BioMedicine in less than 15 hours (with eight A100s)
  arXiv  Detail & Related papers  (2023-06-01T16:50:07Z)
• A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering [2.0559497209595823]
  Research in medical visual question answering (MVQA) can contribute to the development of computer-aided diagnosis. Existing MVQA question extraction schemes mainly focus on word information, ignoring medical information in the text. In this study, a dual-attention learning network with word and sentence embedding (WSDAN) is proposed.
  arXiv  Detail & Related papers  (2022-10-01T08:32:40Z)
• Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge [68.90835997085557]
  We propose a systematic and effective approach to enhance structured medical knowledge from three perspectives. First, we align the representations of the vision encoder and the language encoder through knowledge. Second, we inject knowledge into the multi-modal fusion model to enable the model to perform reasoning using knowledge as the supplementation of the input image and text. Third, we guide the model to put emphasis on the most critical information in images and texts by designing knowledge-induced pretext tasks.
  arXiv  Detail & Related papers  (2022-09-15T08:00:01Z)
• Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training [62.215025958347105]
  We propose a self-supervised learning paradigm with multi-modal masked autoencoders. We learn cross-modal domain knowledge by reconstructing missing pixels and tokens from randomly masked images and texts.
  arXiv  Detail & Related papers  (2022-09-15T07:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.