EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications

• URL: http://arxiv.org/abs/2011.09463v3
• Date: Fri, 20 Aug 2021 07:24:05 GMT
• Title: EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• Authors: Minghui Qiu and Peng Li and Chengyu Wang and Hanjie Pan and Ang Wang and Cen Chen and Xianyan Jia and Yaliang Li and Jun Huang and Deng Cai and Wei Lin
• Abstract summary: EasyTransfer is a platform to develop deep Transfer Learning algorithms for Natural Language Processing (NLP) applications. EasyTransfer supports various NLP models in the ModelZoo, including mainstream PLMs and multi-modality models. EasyTransfer is currently deployed at Alibaba to support a variety of business scenarios.
• Score: 65.87067607849757
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The literature has witnessed the success of leveraging Pre-trained Language Models (PLMs) and Transfer Learning (TL) algorithms to a wide range of Natural Language Processing (NLP) applications, yet it is not easy to build an easy-to-use and scalable TL toolkit for this purpose. To bridge this gap, the EasyTransfer platform is designed to develop deep TL algorithms for NLP applications. EasyTransfer is backended with a high-performance and scalable engine for efficient training and inference, and also integrates comprehensive deep TL algorithms, to make the development of industrial-scale TL applications easier. In EasyTransfer, the built-in data and model parallelism strategies, combined with AI compiler optimization, show to be 4.0x faster than the community version of distributed training. EasyTransfer supports various NLP models in the ModelZoo, including mainstream PLMs and multi-modality models. It also features various in-house developed TL algorithms, together with the AppZoo for NLP applications. The toolkit is convenient for users to quickly start model training, evaluation, and online deployment. EasyTransfer is currently deployed at Alibaba to support a variety of business scenarios, including item recommendation, personalized search, conversational question answering, etc. Extensive experiments on real-world datasets and online applications show that EasyTransfer is suitable for online production with cutting-edge performance for various applications. The source code of EasyTransfer is released at Github (https://github.com/alibaba/EasyTransfer).

Related papers

• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• NVLM: Open Frontier-Class Multimodal LLMs [64.00053046838225]
  We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks. We propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. We develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:59:06Z)
• TransformerRanker: A Tool for Efficiently Finding the Best-Suited Language Models for Downstream Classification Tasks [2.497666465251894]
  TransformerRanker is a lightweight library that ranks pre-trained language models for classification tasks. Our library implements current approaches for transferability estimation. We make TransformerRanker available as a pip-installable open-source library.
  arXiv  Detail & Related papers  (2024-09-09T18:47:00Z)
• Federated Transfer Learning with Task Personalization for Condition Monitoring in Ultrasonic Metal Welding [3.079885946230076]
  This paper presents a Transfer Learning with. Federated Task Task architecture (FTLTP) that provides data capabilities in distributed distributed learning framework. The FTL-TP framework is readily to various other manufacturing applications.
  arXiv  Detail & Related papers  (2024-04-20T05:31:59Z)
• CoLLiE: Collaborative Training of Large Language Models in an Efficient Way [59.09824823710863]
  CoLLiE is an efficient library that facilitates collaborative training of large language models. With its modular design and comprehensive functionality, CoLLiE offers a balanced blend of efficiency, ease of use, and customization.
  arXiv  Detail & Related papers  (2023-12-01T08:02:16Z)
• Simultaneous Machine Translation with Large Language Models [51.470478122113356]
  We investigate the possibility of applying Large Language Models to SimulMT tasks. We conducted experiments using the textttLlama2-7b-chat model on nine different languages from the MUST-C dataset. The results show that LLM outperforms dedicated MT models in terms of BLEU and LAAL metrics.
  arXiv  Detail & Related papers  (2023-09-13T04:06:47Z)
• Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey [0.6240603866868214]
  Multi-Task Learning (MTL) has emerged as a promising approach to improve efficiency and performance through joint training. We discuss the challenges and opportunities of using MTL approaches throughout typical machine learning lifecycle phases. We believe it would be practical to have a model that can handle both MTL and continual learning.
  arXiv  Detail & Related papers  (2023-08-16T09:11:00Z)
• Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction [126.34551436845133]
  CNNs and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL) We present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction.
  arXiv  Detail & Related papers  (2023-08-10T17:37:49Z)
• EasyNLP: A Comprehensive and Easy-to-use Toolkit for Natural Language Processing [38.9428437204642]
  EasyNLP is designed to make it easy to build NLP applications. It features knowledge-enhanced pre-training, knowledge distillation and few-shot learning. EasyNLP has powered over ten business units within Alibaba Group.
  arXiv  Detail & Related papers  (2022-04-30T13:03:53Z)
• On The Cross-Modal Transfer from Natural Language to Code through Adapter Modules [0.0]
  We explore the knowledge transfer using adapters in software engineering. Three programming languages, C/C++, Python, and Java, are studied along with extensive experiments on the best setup used for adapters. Our results can open new directions to build smaller models for more software engineering tasks.
  arXiv  Detail & Related papers  (2022-04-19T04:18:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.