Efficient Methods for Natural Language Processing: A Survey

• URL: http://arxiv.org/abs/2209.00099v2
• Date: Fri, 24 Mar 2023 19:49:14 GMT
• Title: Efficient Methods for Natural Language Processing: A Survey
• Authors: Marcos Treviso, Ji-Ung Lee, Tianchu Ji, Betty van Aken, Qingqing Cao, Manuel R. Ciosici, Michael Hassid, Kenneth Heafield, Sara Hooker, Colin Raffel, Pedro H. Martins, Andr\'e F. T. Martins, Jessica Zosa Forde, Peter Milder, Edwin Simpson, Noam Slonim, Jesse Dodge, Emma Strubell, Niranjan Balasubramanian, Leon Derczynski, Iryna Gurevych, Roy Schwartz
• Abstract summary: This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.
• Score: 76.34572727185896
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

Related papers

• EVOLvE: Evaluating and Optimizing LLMs For Exploration [76.66831821738927]
  Large language models (LLMs) remain under-studied in scenarios requiring optimal decision-making under uncertainty. We measure LLMs' (in)ability to make optimal decisions in bandits, a state-less reinforcement learning setting relevant to many applications. Motivated by the existence of optimal exploration algorithms, we propose efficient ways to integrate this algorithmic knowledge into LLMs.
  arXiv  Detail & Related papers  (2024-10-08T17:54:03Z)
• A Survey on Transformers in NLP with Focus on Efficiency [2.7651063843287718]
  This paper presents a commentary on the evolution of NLP and its applications with emphasis on their accuracy as-well-as efficiency. The goal of this survey is to determine how current NLP techniques contribute towards a sustainable society.
  arXiv  Detail & Related papers  (2024-05-15T10:32:41Z)
• STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models [56.27786433792638]
  STAR is a data generation method that leverages Large Language Models (LLMs) to synthesize data instances. We design fine-grained step-by-step instructions to obtain the initial data instances. Our experiments show that the data generated by STAR significantly improve the performance of low-resource event extraction and relation extraction tasks.
  arXiv  Detail & Related papers  (2023-05-24T12:15:19Z)
• Efficient Exploration using Model-Based Quality-Diversity with Gradients [4.788163807490196]
  In this paper, we propose a model-based Quality-Diversity approach. It extends existing QD methods to use gradients for efficient exploitation and leverage perturbations in imagination for efficient exploration. We demonstrate that it maintains the divergent search capabilities of population-based approaches on tasks with deceptive rewards while significantly improving their sample efficiency and quality of solutions.
  arXiv  Detail & Related papers  (2022-11-22T22:19:01Z)
• High-Resource Methodological Bias in Low-Resource Investigations [27.419604203739052]
  We show that down sampling from a high-resource language results in datasets with different properties than the low-resource datasets. We conclude that naive down sampling of datasets results in a biased view of how well these systems work in a low-resource scenario.
  arXiv  Detail & Related papers  (2022-11-14T17:04:38Z)
• A Survey on Model Compression for Natural Language Processing [13.949219077548687]
  Transformer is preventing NLP from entering broader scenarios including edge and mobile computing. Efficient NLP research aims to comprehensively consider computation, time and carbon emission for the entire life-cycle of NLP.
  arXiv  Detail & Related papers  (2022-02-15T00:18:47Z)
• Efficient Nearest Neighbor Language Models [114.40866461741795]
  Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore. We show how to achieve up to a 6x speed-up in inference speed while retaining comparable performance.
  arXiv  Detail & Related papers  (2021-09-09T12:32:28Z)
• Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
  We propose methods to facilitate identification of training data artifacts. We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data. We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
  arXiv  Detail & Related papers  (2021-07-01T09:26:13Z)
• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP [88.65488361532158]
  dependence on abundant data prevents NLP models from being applied to low-resource settings or novel tasks. Data augmentation methods have been explored as a means of improving data efficiency in NLP. We provide an empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting.
  arXiv  Detail & Related papers  (2021-06-14T15:27:22Z)
• A Little Pretraining Goes a Long Way: A Case Study on Dependency Parsing Task for Low-resource Morphologically Rich Languages [14.694800341598368]
  We focus on dependency parsing for morphological rich languages (MRLs) in a low-resource setting. To address these challenges, we propose simple auxiliary tasks for pretraining. We perform experiments on 10 MRLs in low-resource settings to measure the efficacy of our proposed pretraining method.
  arXiv  Detail & Related papers  (2021-02-12T14:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.