LiteMuL: A Lightweight On-Device Sequence Tagger using Multi-task Learning

• URL: http://arxiv.org/abs/2101.03024v2
• Date: Mon, 29 Mar 2021 14:31:19 GMT
• Title: LiteMuL: A Lightweight On-Device Sequence Tagger using Multi-task Learning
• Authors: Sonal Kumari, Vibhav Agarwal, Bharath Challa, Kranti Chalamalasetti, Sourav Ghosh, Harshavardhana, Barath Raj Kandur Raja
• Abstract summary: LiteMuL is a lightweight on-device sequence tagger that can efficiently process the user conversations using a Multi-Task Learning approach. Our model is competitive with other MTL approaches for NER and POS tasks while outshines them with a low memory footprint.
• Score: 1.3192560874022086
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Named entity detection and Parts-of-speech tagging are the key tasks for many NLP applications. Although the current state of the art methods achieved near perfection for long, formal, structured text there are hindrances in deploying these models on memory-constrained devices such as mobile phones. Furthermore, the performance of these models is degraded when they encounter short, informal, and casual conversations. To overcome these difficulties, we present LiteMuL - a lightweight on-device sequence tagger that can efficiently process the user conversations using a Multi-Task Learning (MTL) approach. To the best of our knowledge, the proposed model is the first on-device MTL neural model for sequence tagging. Our LiteMuL model is about 2.39 MB in size and achieved an accuracy of 0.9433 (for NER), 0.9090 (for POS) on the CoNLL 2003 dataset. The proposed LiteMuL not only outperforms the current state of the art results but also surpasses the results of our proposed on-device task-specific models, with accuracy gains of up to 11% and model-size reduction by 50%-56%. Our model is competitive with other MTL approaches for NER and POS tasks while outshines them with a low memory footprint. We also evaluated our model on custom-curated user conversations and observed impressive results.

Related papers

• MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic [6.46176287368784]
  We propose textbfModel textbfExclusive textbfTask textbfArithmetic for merging textbfGPT-scale models. Our proposed MetaGPT is data-agnostic and bypasses the heavy search process, making it cost-effective and easy to implement for LLMs.
  arXiv  Detail & Related papers  (2024-06-17T10:12:45Z)
• Herd: Using multiple, smaller LLMs to match the performances of proprietary, large LLMs via an intelligent composer [1.3108652488669732]
  We show that a herd of open source models can match or exceed the performance of proprietary models via an intelligent router. In cases where GPT is not able to answer the query, Herd is able to identify a model that can, at least 40% of the time.
  arXiv  Detail & Related papers  (2023-10-30T18:11:02Z)
• Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes [91.58845026796149]
  We introduce Distilling step-by-step, a new mechanism that trains small models that outperform large language models. We present three findings across 4 NLP benchmarks.
  arXiv  Detail & Related papers  (2023-05-03T17:50:56Z)
• AdaMTL: Adaptive Input-dependent Inference for Efficient Multi-Task Learning [1.4963011898406864]
  We introduce AdaMTL, an adaptive framework that learns task-aware inference policies for multi-task learning models. AdaMTL reduces the computational complexity by 43% while improving the accuracy by 1.32% compared to single-task models. When deployed on Vuzix M4000 smart glasses, AdaMTL reduces the inference latency and the energy consumption by up to 21.8% and 37.5%, respectively.
  arXiv  Detail & Related papers  (2023-04-17T20:17:44Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)
• Incremental Online Learning Algorithms Comparison for Gesture and Visual Smart Sensors [68.8204255655161]
  This paper compares four state-of-the-art algorithms in two real applications: gesture recognition based on accelerometer data and image classification. Our results confirm these systems' reliability and the feasibility of deploying them in tiny-memory MCUs.
  arXiv  Detail & Related papers  (2022-09-01T17:05:20Z)
• PaLM: Scaling Language Modeling with Pathways [180.69584031908113]
  We trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2022-04-05T16:11:45Z)
• pNLP-Mixer: an Efficient all-MLP Architecture for Language [10.634940525287014]
  pNLP-Mixer model for on-device NLP achieves high weight-efficiency thanks to a novel projection layer. We evaluate a pNLP-Mixer model of only one megabyte in size on two multi-lingual semantic parsing datasets, MTOP and multiATIS. Our model consistently beats the state-of-the-art of tiny models, which is twice as large, by a margin up to 7.8% on MTOP.
  arXiv  Detail & Related papers  (2022-02-09T09:01:29Z)
• LiST: Lite Self-training Makes Efficient Few-shot Learners [91.28065455714018]
  LiST improves by 35% over classic fine-tuning methods and 6% over prompt-tuning with 96% reduction in number of trainable parameters when fine-tuned with no more than 30 labeled examples from each target domain.
  arXiv  Detail & Related papers  (2021-10-12T18:47:18Z)
• MoEfication: Conditional Computation of Transformer Models for Efficient Inference [66.56994436947441]
  Transformer-based pre-trained language models can achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. We explore to accelerate large-model inference by conditional computation based on the sparse activation phenomenon. We propose to transform a large model into its mixture-of-experts (MoE) version with equal model size, namely MoEfication.
  arXiv  Detail & Related papers  (2021-10-05T02:14:38Z)
• Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data [5.689320790746046]
  Multi-Task Learning (MTL) networks have emerged as a promising method for transferring learned knowledge across different tasks. However, MTL must deal with challenges such as: overfitting to low resource tasks, catastrophic forgetting, and negative task transfer. We propose a novel Transformer architecture consisting of a new conditional attention mechanism and a set of task-conditioned modules.
  arXiv  Detail & Related papers  (2020-09-19T02:04:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.