On-edge Multi-task Transfer Learning: Model and Practice with
Data-driven Task Allocation
- URL: http://arxiv.org/abs/2107.02466v1
- Date: Tue, 6 Jul 2021 08:24:25 GMT
- Title: On-edge Multi-task Transfer Learning: Model and Practice with
Data-driven Task Allocation
- Authors: Zimu Zheng, Qiong Chen, Chuang Hu, Dan Wang, Fangming Liu
- Abstract summary: We show that task allocation with task importance for Multi-task Transfer Learning (MTL) is a variant of the NP-complete Knapsack problem.
We propose a Data-driven Cooperative Task Allocation (DCTA) approach to solve TATIM with high computational efficiency.
Our DCTA reduces 3.24 times of processing time, and saves 48.4% energy consumption compared with the state-of-the-art when solving TATIM.
- Score: 20.20889051697198
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: On edge devices, data scarcity occurs as a common problem where transfer
learning serves as a widely-suggested remedy. Nevertheless, transfer learning
imposes a heavy computation burden to resource-constrained edge devices.
Existing task allocation works usually assume all submitted tasks are equally
important, leading to inefficient resource allocation at a task level when
directly applied in Multi-task Transfer Learning (MTL). To address these
issues, we first reveal that it is crucial to measure the impact of tasks on
overall decision performance improvement and quantify \emph{task importance}.
We then show that task allocation with task importance for MTL (TATIM) is a
variant of the NP-complete Knapsack problem, where the complicated computation
to solve this problem needs to be conducted repeatedly under varying contexts.
To solve TATIM with high computational efficiency, we propose a Data-driven
Cooperative Task Allocation (DCTA) approach. Finally, we evaluate the
performance of DCTA by not only a trace-driven simulation, but also a new
comprehensive real-world AIOps case study that bridges model and practice via a
new architecture and main components design within the AIOps system. Extensive
experiments show that our DCTA reduces 3.24 times of processing time, and saves
48.4\% energy consumption compared with the state-of-the-art when solving
TATIM.
Related papers
- InterroGate: Learning to Share, Specialize, and Prune Representations
for Multi-task Learning [17.66308231838553]
We propose a novel multi-task learning (MTL) architecture designed to mitigate task interference while optimizing inference computational efficiency.
We employ a learnable gating mechanism to automatically balance the shared and task-specific representations while preserving the performance of all tasks.
arXiv Detail & Related papers (2024-02-26T18:59:52Z) - Data-CUBE: Data Curriculum for Instruction-based Sentence Representation
Learning [85.66907881270785]
We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training.
In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk.
In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
arXiv Detail & Related papers (2024-01-07T18:12:20Z) - Distribution Matching for Multi-Task Learning of Classification Tasks: a
Large-Scale Study on Faces & Beyond [62.406687088097605]
Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space.
We show that MTL can be successful with classification tasks with little, or non-overlapping annotations.
We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
arXiv Detail & Related papers (2024-01-02T14:18:11Z) - Task Aware Feature Extraction Framework for Sequential Dependence
Multi-Task Learning [1.0765359420035392]
We analyze sequential dependence MTL from rigorous mathematical perspective.
We propose a Task Aware Feature Extraction (TAFE) framework for sequential dependence MTL.
arXiv Detail & Related papers (2023-01-06T13:12:59Z) - Active Multi-Task Representation Learning [50.13453053304159]
We give the first formal study on resource task sampling by leveraging the techniques from active learning.
We propose an algorithm that iteratively estimates the relevance of each source task to the target task and samples from each source task based on the estimated relevance.
arXiv Detail & Related papers (2022-02-02T08:23:24Z) - Transfer Learning in Conversational Analysis through Reusing
Preprocessing Data as Supervisors [52.37504333689262]
Using noisy labels in single-task learning increases the risk of over-fitting.
Auxiliary tasks could improve the performance of the primary task learning during the same training.
arXiv Detail & Related papers (2021-12-02T08:40:42Z) - Understanding and Improving Information Transfer in Multi-Task Learning [14.43111978531182]
We study an architecture with a shared module for all tasks and a separate output module for each task.
We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer.
Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning.
arXiv Detail & Related papers (2020-05-02T23:43:52Z) - Exploring and Predicting Transferability across NLP Tasks [115.6278033699853]
We study the transferability between 33 NLP tasks across three broad classes of problems.
Our results show that transfer learning is more beneficial than previously thought.
We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task.
arXiv Detail & Related papers (2020-05-02T09:39:36Z) - Emerging Relation Network and Task Embedding for Multi-Task Regression
Problems [5.953831950062808]
Multi-task learning (mtl) provides state-of-the-art results in many applications of computer vision and natural language processing.
This article provides a comparative study of the following recent and important mtl architectures.
We introduce a new mtl architecture named emerging relation network (ern) which can be considered as an extension of the sluice network.
arXiv Detail & Related papers (2020-04-29T09:02:24Z) - Task-Feature Collaborative Learning with Application to Personalized
Attribute Prediction [166.87111665908333]
We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL)
Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks.
As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
arXiv Detail & Related papers (2020-04-29T02:32:04Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.