Modular Approach to Machine Reading Comprehension: Mixture of Task-Aware Experts

• URL: http://arxiv.org/abs/2210.01750v1
• Date: Tue, 4 Oct 2022 17:13:41 GMT
• Title: Modular Approach to Machine Reading Comprehension: Mixture of Task-Aware Experts
• Authors: Anirudha Rayasam, Anusha Kamath, Gabriel Bayomi Tinoco Kalejaiye
• Abstract summary: We present a Mixture of Task-Aware Experts Network for Machine Reading on a relatively small dataset. We focus on the issue of common-sense learning, enforcing the common ground knowledge. We take inspi ration on the recent advancements of multitask and transfer learning.
• Score: 0.5801044612920815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work we present a Mixture of Task-Aware Experts Network for Machine Reading Comprehension on a relatively small dataset. We particularly focus on the issue of common-sense learning, enforcing the common ground knowledge by specifically training different expert networks to capture different kinds of relationships between each passage, question and choice triplet. Moreover, we take inspi ration on the recent advancements of multitask and transfer learning by training each network a relevant focused task. By making the mixture-of-networks aware of a specific goal by enforcing a task and a relationship, we achieve state-of-the-art results and reduce over-fitting.

Related papers

• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• OmniVec: Learning robust representations with cross modal sharing [28.023214572340336]
  We present an approach to learn multiple tasks, in multiple modalities, with a unified architecture. The proposed network is composed of task specific encoders, a common trunk in the middle, followed by task specific prediction heads. We train the network on all major modalities, e.g. visual, audio, text and 3D, and report results on $22$ diverse and challenging public benchmarks.
  arXiv  Detail & Related papers  (2023-11-07T14:00:09Z)
• Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding [52.723297744257536]
  Pre-trained language models (LMs) have shown effectiveness in scientific literature understanding tasks. We propose a multi-task contrastive learning framework, SciMult, to facilitate common knowledge sharing across different literature understanding tasks.
  arXiv  Detail & Related papers  (2023-05-23T16:47:22Z)
• Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners [67.5865966762559]
  We study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning. We devise task-aware gating functions to route examples from different tasks to specialized experts. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model.
  arXiv  Detail & Related papers  (2022-04-16T00:56:12Z)
• Learning Multi-Tasks with Inconsistent Labels by using Auxiliary Big Task [24.618094251341958]
  Multi-task learning is to improve the performance of the model by transferring and exploiting common knowledge among tasks. We propose a framework to learn these tasks by jointly leveraging both abundant information from a learnt auxiliary big task with sufficiently many classes to cover those of all these tasks. Our experimental results demonstrate its effectiveness in comparison with the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-07T02:46:47Z)
• On the relationship between disentanglement and multi-task learning [62.997667081978825]
  We take a closer look at the relationship between disentanglement and multi-task learning based on hard parameter sharing. We show that disentanglement appears naturally during the process of multi-task neural network training.
  arXiv  Detail & Related papers  (2021-10-07T14:35:34Z)
• Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study [75.42182503265056]
  Multi-Task Learning has emerged as a methodology in which multiple tasks are jointly learned by a shared learning algorithm. We deal with heterogeneous MTL, simultaneously addressing detection, classification & regression problems. We build FaceBehaviorNet, the first framework for large-scale face analysis, by jointly learning all facial behavior tasks.
  arXiv  Detail & Related papers  (2021-05-08T22:26:52Z)
• Multi-Task Learning with Deep Neural Networks: A Survey [0.0]
  Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. We give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field.
  arXiv  Detail & Related papers  (2020-09-10T19:31:04Z)
• Navigating the Trade-Off between Multi-Task Learning and Learning to Multitask in Deep Neural Networks [9.278739724750343]
  Multi-task learning refers to a paradigm in machine learning in which a network is trained on various related tasks to facilitate the acquisition of tasks. multitasking is used to indicate, especially in the cognitive science literature, the ability to execute multiple tasks simultaneously. We show that the same tension arises in deep networks and discuss a meta-learning algorithm for an agent to manage this trade-off in an unfamiliar environment.
  arXiv  Detail & Related papers  (2020-07-20T23:26:16Z)
• Auxiliary Learning by Implicit Differentiation [54.92146615836611]
  Training neural networks with auxiliary tasks is a common practice for improving the performance on a main task of interest. Here, we propose a novel framework, AuxiLearn, that targets both challenges based on implicit differentiation. First, when useful auxiliaries are known, we propose learning a network that combines all losses into a single coherent objective function. Second, when no useful auxiliary task is known, we describe how to learn a network that generates a meaningful, novel auxiliary task.
  arXiv  Detail & Related papers  (2020-06-22T19:35:07Z)
• Multitask learning over graphs: An Approach for Distributed, Streaming Machine Learning [46.613346075513206]
  Multitask learning is an approach to inductive transfer learning. Recent years have witnessed an increasing ability to collect data in a distributed and streaming manner. This requires the design of new strategies for learning jointly multiple tasks from streaming data over distributed (or networked) systems.
  arXiv  Detail & Related papers  (2020-01-07T15:32:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.