Building an Efficiency Pipeline: Commutativity and Cumulativeness of
Efficiency Operators for Transformers
- URL: http://arxiv.org/abs/2208.00483v1
- Date: Sun, 31 Jul 2022 18:01:06 GMT
- Title: Building an Efficiency Pipeline: Commutativity and Cumulativeness of
Efficiency Operators for Transformers
- Authors: Ji Xin, Raphael Tang, Zhiying Jiang, Yaoliang Yu, Jimmy Lin
- Abstract summary: We consider an efficiency method as an operator applied on a model.
In this paper, we study the plausibility of this idea, and the commutativity and cumulativeness of efficiency operators.
- Score: 68.55472265775514
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: There exists a wide variety of efficiency methods for natural language
processing (NLP) tasks, such as pruning, distillation, dynamic inference,
quantization, etc. We can consider an efficiency method as an operator applied
on a model. Naturally, we may construct a pipeline of multiple efficiency
methods, i.e., to apply multiple operators on the model sequentially. In this
paper, we study the plausibility of this idea, and more importantly, the
commutativity and cumulativeness of efficiency operators. We make two
interesting observations: (1) Efficiency operators are commutative -- the order
of efficiency methods within the pipeline has little impact on the final
results; (2) Efficiency operators are also cumulative -- the final results of
combining several efficiency methods can be estimated by combining the results
of individual methods. These observations deepen our understanding of
efficiency operators and provide useful guidelines for their real-world
applications.
Related papers
- Diffusion Models as Optimizers for Efficient Planning in Offline RL [47.0835433289033]
Diffusion models have shown strong competitiveness in offline reinforcement learning tasks.
We propose a faster autoregressive model to handle the generation of feasible trajectories.
This allows us to achieve more efficient planning without sacrificing capability.
arXiv Detail & Related papers (2024-07-23T03:00:01Z) - Efficiency fluctuations of a heat engine with noise induced quantum coherences [0.0]
We analyze the efficiency fluctuations of a coherent quantum heat engine coupled to a unimodal cavity.
The most likely efficiency is independent of the noise induced coherences.
Treating the efficiency as a variable allows the enhancement of constancy only in the presence of coherences.
arXiv Detail & Related papers (2024-05-29T14:55:25Z) - ECO-TR: Efficient Correspondences Finding Via Coarse-to-Fine Refinement [80.94378602238432]
We propose an efficient structure named Correspondence Efficient Transformer (ECO-TR) by finding correspondences in a coarse-to-fine manner.
To achieve this, multiple transformer blocks are stage-wisely connected to gradually refine the predicted coordinates.
Experiments on various sparse and dense matching tasks demonstrate the superiority of our method in both efficiency and effectiveness against existing state-of-the-arts.
arXiv Detail & Related papers (2022-09-25T13:05:33Z) - Automated Evolutionary Approach for the Design of Composite Machine
Learning Pipelines [48.7576911714538]
The proposed approach is aimed to automate the design of composite machine learning pipelines.
It designs the pipelines with a customizable graph-based structure, analyzes the obtained results, and reproduces them.
The software implementation on this approach is presented as an open-source framework.
arXiv Detail & Related papers (2021-06-26T23:19:06Z) - Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
Learning from implicit feedback is challenging because of the difficult nature of the one-class problem.
Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem.
We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
arXiv Detail & Related papers (2021-05-11T03:38:16Z) - Woodbury Transformations for Deep Generative Flows [17.062207075794205]
We introduce Woodbury transformations, which achieve efficient invertibility via the Woodbury matrix identity.
Woodbury transformations enable (1) high-dimensional interactions, (2) efficient sampling, and (3) efficient likelihood evaluation.
arXiv Detail & Related papers (2020-02-27T16:21:43Z) - Learning with Differentiable Perturbed Optimizers [54.351317101356614]
We propose a systematic method to transform operations into operations that are differentiable and never locally constant.
Our approach relies on perturbeds, and can be used readily together with existing solvers.
We show how this framework can be connected to a family of losses developed in structured prediction, and give theoretical guarantees for their use in learning tasks.
arXiv Detail & Related papers (2020-02-20T11:11:32Z) - HULK: An Energy Efficiency Benchmark Platform for Responsible Natural
Language Processing [76.38975568873765]
We introduce HULK, a multi-task energy efficiency benchmarking platform for responsible natural language processing.
We compare pretrained models' energy efficiency from the perspectives of time and cost.
arXiv Detail & Related papers (2020-02-14T01:04:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.