Single-Modal Entropy based Active Learning for Visual Question Answering

• URL: http://arxiv.org/abs/2110.10906v1
• Date: Thu, 21 Oct 2021 05:38:45 GMT
• Title: Single-Modal Entropy based Active Learning for Visual Question Answering
• Authors: Dong-Jin Kim, Jae Won Cho, Jinsoo Choi, Yunjae Jung, In So Kweon
• Abstract summary: We address Active Learning in the multi-modal setting of Visual Question Answering (VQA) In light of the multi-modal inputs, image and question, we propose a novel method for effective sample acquisition. Our novel idea is simple to implement, cost-efficient, and readily adaptable to other multi-modal tasks.
• Score: 75.1682163844354
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Constructing a large-scale labeled dataset in the real world, especially for high-level tasks (eg, Visual Question Answering), can be expensive and time-consuming. In addition, with the ever-growing amounts of data and architecture complexity, Active Learning has become an important aspect of computer vision research. In this work, we address Active Learning in the multi-modal setting of Visual Question Answering (VQA). In light of the multi-modal inputs, image and question, we propose a novel method for effective sample acquisition through the use of ad hoc single-modal branches for each input to leverage its information. Our mutual information based sample acquisition strategy Single-Modal Entropic Measure (SMEM) in addition to our self-distillation technique enables the sample acquisitor to exploit all present modalities and find the most informative samples. Our novel idea is simple to implement, cost-efficient, and readily adaptable to other multi-modal tasks. We confirm our findings on various VQA datasets through state-of-the-art performance by comparing to existing Active Learning baselines.

Related papers

• Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors [10.454194186065195]
  Reinforcement learning has achieved promising results on robotic control tasks but struggles to leverage information effectively. Recent works construct auxiliary losses based on reconstruction or mutual information to extract joint representations from multiple sensory inputs. We argue that compressing information in the learned joint representations about raw multimodal observations is helpful.
  arXiv  Detail & Related papers  (2024-10-23T04:32:37Z)
• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Learning Unseen Modality Interaction [54.23533023883659]
  Multimodal learning assumes all modality combinations of interest are available during training to learn cross-modal correspondences. We pose the problem of unseen modality interaction and introduce a first solution. It exploits a module that projects the multidimensional features of different modalities into a common space with rich information preserved.
  arXiv  Detail & Related papers  (2023-06-22T10:53:10Z)
• Multi-View Class Incremental Learning [57.14644913531313]
  Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance. This paper investigates a novel paradigm called multi-view class incremental learning (MVCIL), where a single model incrementally classifies new classes from a continual stream of views.
  arXiv  Detail & Related papers  (2023-06-16T08:13:41Z)
• Factorized Contrastive Learning: Going Beyond Multi-view Redundancy [116.25342513407173]
  This paper proposes FactorCL, a new multimodal representation learning method to go beyond multi-view redundancy. On large-scale real-world datasets, FactorCL captures both shared and unique information and achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2023-06-08T15:17:04Z)
• Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications [47.501121601856795]
  Multimodality Representation Learning is a technique of learning to embed information from different modalities and their correlations. Cross-modal interaction and complementary information from different modalities are crucial for advanced models to perform any multimodal task. This survey presents the literature on the evolution and enhancement of deep learning multimodal architectures.
  arXiv  Detail & Related papers  (2023-02-01T11:48:34Z)
• Contrastive Learning with Cross-Modal Knowledge Mining for Multimodal Human Activity Recognition [1.869225486385596]
  We explore the hypothesis that leveraging multiple modalities can lead to better recognition. We extend a number of recent contrastive self-supervised approaches for the task of Human Activity Recognition. We propose a flexible, general-purpose framework for performing multimodal self-supervised learning.
  arXiv  Detail & Related papers  (2022-05-20T10:39:16Z)
• Active Multi-Task Representation Learning [50.13453053304159]
  We give the first formal study on resource task sampling by leveraging the techniques from active learning. We propose an algorithm that iteratively estimates the relevance of each source task to the target task and samples from each source task based on the estimated relevance.
  arXiv  Detail & Related papers  (2022-02-02T08:23:24Z)
• Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis [11.368438990334397]
  We develop a self-supervised learning strategy to acquire independent unimodal supervisions. We conduct extensive experiments on three public multimodal baseline datasets. Our method achieves comparable performance than human-annotated unimodal labels.
  arXiv  Detail & Related papers  (2021-02-09T14:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.