Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis

• URL: http://arxiv.org/abs/2406.15050v1
• Date: Fri, 21 Jun 2024 10:50:55 GMT
• Title: Tri-VQA: Triangular Reasoning Medical Visual Question Answering for Multi-Attribute Analysis
• Authors: Lin Fan, Xun Gong, Cenyang Zheng, Yafei Ou,
• Abstract summary: We investigate the construction of a more cohesive and stable Med-VQA structure. Motivated by causal effect, we propose a novel Triangular Reasoning VQA framework.
• Score: 4.964280449393689
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The intersection of medical Visual Question Answering (Med-VQA) is a challenging research topic with advantages including patient engagement and clinical expert involvement for second opinions. However, existing Med-VQA methods based on joint embedding fail to explain whether their provided results are based on correct reasoning or coincidental answers, which undermines the credibility of VQA answers. In this paper, we investigate the construction of a more cohesive and stable Med-VQA structure. Motivated by causal effect, we propose a novel Triangular Reasoning VQA (Tri-VQA) framework, which constructs reverse causal questions from the perspective of "Why this answer?" to elucidate the source of the answer and stimulate more reasonable forward reasoning processes. We evaluate our method on the Endoscopic Ultrasound (EUS) multi-attribute annotated dataset from five centers, and test it on medical VQA datasets. Experimental results demonstrate the superiority of our approach over existing methods. Our codes and pre-trained models are available at https://anonymous.4open.science/r/Tri_VQA.

Related papers

• Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidence [33.018873142559286]
  We propose a novel approach for generating natural language explanations for answers predicted by medical QA systems. Our system extract knowledge from medical textbooks to enhance the quality of explanations during the explanation generation process.
  arXiv  Detail & Related papers  (2023-10-02T16:00:37Z)
• Visual Question Answering in the Medical Domain [13.673890873313354]
  We present a novel contrastive learning pretraining method to mitigate the problem of small datasets for the Med-VQA task. Our proposed model obtained an accuracy of 60% on the VQA-Med 2019 test set, giving comparable results to other state-of-the-art Med-VQA models.
  arXiv  Detail & Related papers  (2023-09-20T06:06:10Z)
• PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering [56.25766322554655]
  Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery. We propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model. We train the proposed model on PMC-VQA and then fine-tune it on multiple public benchmarks, e.g., VQA-RAD, SLAKE, and Image-Clef 2019.
  arXiv  Detail & Related papers  (2023-05-17T17:50:16Z)
• Consistency-preserving Visual Question Answering in Medical Imaging [2.005299372367689]
  Visual Question Answering (VQA) models take an image and a natural-language question as input and infer the answer to the question. We propose a novel loss function and corresponding training procedure that allows the inclusion of relations between questions into the training process. Our experiments show that our method outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-06-27T13:38:50Z)
• Learning to Answer Questions in Dynamic Audio-Visual Scenarios [81.19017026999218]
  We focus on the Audio-Visual Questioning (AVQA) task, which aims to answer questions regarding different visual objects sounds, and their associations in videos. Our dataset contains more than 45K question-answer pairs spanning over different modalities and question types. Our results demonstrate that AVQA benefits from multisensory perception and our model outperforms recent A-SIC, V-SIC, and AVQA approaches.
  arXiv  Detail & Related papers  (2022-03-26T13:03:42Z)
• Medical Visual Question Answering: A Survey [55.53205317089564]
  Medical Visual Question Answering(VQA) is a combination of medical artificial intelligence and popular VQA challenges. Given a medical image and a clinically relevant question in natural language, the medical VQA system is expected to predict a plausible and convincing answer.
  arXiv  Detail & Related papers  (2021-11-19T05:55:15Z)
• Human-Adversarial Visual Question Answering [62.30715496829321]
  We benchmark state-of-the-art VQA models against human-adversarial examples. We find that a wide range of state-of-the-art models perform poorly when evaluated on these examples.
  arXiv  Detail & Related papers  (2021-06-04T06:25:32Z)
• Hierarchical Deep Multi-modal Network for Medical Visual Question Answering [25.633660028022195]
  We propose a hierarchical deep multi-modal network that analyzes and classifies end-user questions/queries. We integrate the QS model to the hierarchical deep multi-modal neural network to generate proper answers to the queries related to medical images.
  arXiv  Detail & Related papers  (2020-09-27T07:24:41Z)
• Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering [89.76059961309453]
  HeadQA dataset contains multiple-choice questions authorized for the public healthcare specialization exam. These questions are the most challenging for current QA systems. We present a Multi-step reasoning with Knowledge extraction framework (MurKe) We are striving to make full use of off-the-shelf pre-trained models.
  arXiv  Detail & Related papers  (2020-08-06T02:47:46Z)
• SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions [66.86887670416193]
  We show that state-of-the-art VQA models have comparable performance in answering perception and reasoning questions, but suffer from consistency problems. To address this shortcoming, we propose an approach called Sub-Question-aware Network Tuning (SQuINT) We show that SQuINT improves model consistency by 5%, also marginally improving performance on the Reasoning questions in VQA, while also displaying better attention maps.
  arXiv  Detail & Related papers  (2020-01-20T01:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.