Building an Efficiency Pipeline: Commutativity and Cumulativeness of Efficiency Operators for Transformers

• URL: http://arxiv.org/abs/2208.00483v1
• Date: Sun, 31 Jul 2022 18:01:06 GMT
• Title: Building an Efficiency Pipeline: Commutativity and Cumulativeness of Efficiency Operators for Transformers
• Authors: Ji Xin, Raphael Tang, Zhiying Jiang, Yaoliang Yu, Jimmy Lin
• Abstract summary: We consider an efficiency method as an operator applied on a model. In this paper, we study the plausibility of this idea, and the commutativity and cumulativeness of efficiency operators.
• Score: 68.55472265775514
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There exists a wide variety of efficiency methods for natural language processing (NLP) tasks, such as pruning, distillation, dynamic inference, quantization, etc. We can consider an efficiency method as an operator applied on a model. Naturally, we may construct a pipeline of multiple efficiency methods, i.e., to apply multiple operators on the model sequentially. In this paper, we study the plausibility of this idea, and more importantly, the commutativity and cumulativeness of efficiency operators. We make two interesting observations: (1) Efficiency operators are commutative -- the order of efficiency methods within the pipeline has little impact on the final results; (2) Efficiency operators are also cumulative -- the final results of combining several efficiency methods can be estimated by combining the results of individual methods. These observations deepen our understanding of efficiency operators and provide useful guidelines for their real-world applications.

Related papers

• DELIFT: Data Efficient Language model Instruction Fine Tuning [6.425903493706273]
  We introduce DELIFT, a novel algorithm that systematically optimize data selection across the three key stages of fine-tuning. Experiments across various tasks and model scales demonstrate that DELIFT can reduce the fine-tuning data size by up to 70% without compromising performance.
  arXiv  Detail & Related papers  (2024-11-07T04:38:29Z)
• Faster WIND: Accelerating Iterative Best-of-$N$ Distillation for LLM Alignment [81.84950252537618]
  This paper reveals a unified game-theoretic connection between iterative BOND and self-play alignment. We establish a novel framework, WIN rate Dominance (WIND), with a series of efficient algorithms for regularized win rate dominance optimization.
  arXiv  Detail & Related papers  (2024-10-28T04:47:39Z)
• Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation [69.60321475454843]
  We propose DPCPL, the first pre-training and prompt-tuning paradigm tailored for Multi-Behavior Sequential Recommendation. In the pre-training stage, we propose a novel Efficient Behavior Miner (EBM) to filter out the noise at multiple time scales. Subsequently, we propose to tune the pre-trained model in a highly efficient manner with the proposed Customized Prompt Learning (CPL) module.
  arXiv  Detail & Related papers  (2024-08-21T06:48:38Z)
• Evaluating Language Models for Efficient Code Generation [13.175840119811]
  We introduce Differential Performance Evaluation (DPE) to reliably evaluate Large Language Models (LLMs) DPE focuses on efficiency-demanding programming tasks and establishing an insightful compound metric for performance evaluation. As a proof of concept, we use DPE to create EvalPerf, a benchmark with 121 performance-challenging coding tasks.
  arXiv  Detail & Related papers  (2024-08-12T18:59:13Z)
• ECO-TR: Efficient Correspondences Finding Via Coarse-to-Fine Refinement [80.94378602238432]
  We propose an efficient structure named Correspondence Efficient Transformer (ECO-TR) by finding correspondences in a coarse-to-fine manner. To achieve this, multiple transformer blocks are stage-wisely connected to gradually refine the predicted coordinates. Experiments on various sparse and dense matching tasks demonstrate the superiority of our method in both efficiency and effectiveness against existing state-of-the-arts.
  arXiv  Detail & Related papers  (2022-09-25T13:05:33Z)
• Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelines [48.7576911714538]
  The proposed approach is aimed to automate the design of composite machine learning pipelines. It designs the pipelines with a customizable graph-based structure, analyzes the obtained results, and reproduces them. The software implementation on this approach is presented as an open-source framework.
  arXiv  Detail & Related papers  (2021-06-26T23:19:06Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• Woodbury Transformations for Deep Generative Flows [17.062207075794205]
  We introduce Woodbury transformations, which achieve efficient invertibility via the Woodbury matrix identity. Woodbury transformations enable (1) high-dimensional interactions, (2) efficient sampling, and (3) efficient likelihood evaluation.
  arXiv  Detail & Related papers  (2020-02-27T16:21:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.