Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research

• URL: http://arxiv.org/abs/2306.16900v2
• Date: Thu, 9 Nov 2023 13:52:45 GMT
• Title: Surveying (Dis)Parities and Concerns of Compute Hungry NLP Research
• Authors: Ji-Ung Lee, Haritz Puerto, Betty van Aken, Yuki Arase, Jessica Zosa Forde, Leon Derczynski, Andreas R\"uckl\'e, Iryna Gurevych, Roy Schwartz, Emma Strubell, Jesse Dodge
• Abstract summary: We provide a first attempt to quantify concerns regarding three topics, namely, environmental impact, equity, and impact on peer reviewing. We capture existing (dis)parities between different and within groups with respect to seniority, academia, and industry. We devise recommendations to mitigate found disparities, some of which already successfully implemented.
• Score: 75.84463664853125
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many recent improvements in NLP stem from the development and use of large pre-trained language models (PLMs) with billions of parameters. Large model sizes makes computational cost one of the main limiting factors for training and evaluating such models; and has raised severe concerns about the sustainability, reproducibility, and inclusiveness for researching PLMs. These concerns are often based on personal experiences and observations. However, there had not been any large-scale surveys that investigate them. In this work, we provide a first attempt to quantify these concerns regarding three topics, namely, environmental impact, equity, and impact on peer reviewing. By conducting a survey with 312 participants from the NLP community, we capture existing (dis)parities between different and within groups with respect to seniority, academia, and industry; and their impact on the peer reviewing process. For each topic, we provide an analysis and devise recommendations to mitigate found disparities, some of which already successfully implemented. Finally, we discuss additional concerns raised by many participants in free-text responses.

Related papers

• On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs [20.589396689900614]
  This paper addresses three fundamental questions: Why do we need interpretability, what are we interpreting, and how? By exploring these questions, we examine existing interpretability paradigms, their properties, and their relevance to different stakeholders. Our analysis reveals significant disparities between NLP developers and non-developer users, as well as between research fields, underscoring the diverse needs of stakeholders.
  arXiv  Detail & Related papers  (2024-07-27T08:00:27Z)
• From Insights to Actions: The Impact of Interpretability and Analysis Research on NLP [28.942812379900673]
  Interpretability and analysis (IA) research is a growing subfield within NLP. We seek to quantify the impact of IA research on the broader field of NLP.
  arXiv  Detail & Related papers  (2024-06-18T13:45:07Z)
• What Can Natural Language Processing Do for Peer Review? [173.8912784451817]
  In modern science, peer review is widely used, yet it is hard, time-consuming, and prone to error. Since the artifacts involved in peer review are largely text-based, Natural Language Processing has great potential to improve reviewing. We detail each step of the process from manuscript submission to camera-ready revision, and discuss the associated challenges and opportunities for NLP assistance.
  arXiv  Detail & Related papers  (2024-05-10T16:06:43Z)
• Responsible AI Considerations in Text Summarization Research: A Review of Current Practices [89.85174013619883]
  We focus on text summarization, a common NLP task largely overlooked by the responsible AI community. We conduct a multi-round qualitative analysis of 333 summarization papers from the ACL Anthology published between 2020-2022. We focus on how, which, and when responsible AI issues are covered, which relevant stakeholders are considered, and mismatches between stated and realized research goals.
  arXiv  Detail & Related papers  (2023-11-18T15:35:36Z)
• SemEval-2023 Task 11: Learning With Disagreements (LeWiDi) [75.85548747729466]
  We report on the second edition of the LeWiDi series of shared tasks. This second edition attracted a wide array of participants resulting in 13 shared task submission papers.
  arXiv  Detail & Related papers  (2023-04-28T12:20:35Z)
• A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing [68.37496795076203]
  We provide guidance for NLP researchers and practitioners dealing with imbalanced data. We first discuss various types of controlled and real-world class imbalance. We organize the methods by whether they are based on sampling, data augmentation, choice of loss function, staged learning, or model design.
  arXiv  Detail & Related papers  (2022-10-10T13:26:40Z)
• What Do NLP Researchers Believe? Results of the NLP Community Metasurvey [43.763865378178245]
  We present the results of the NLP Community Metasurvey. The survey elicited opinions on controversial issues. We find false sociological beliefs where the community's predictions don't match reality.
  arXiv  Detail & Related papers  (2022-08-26T19:45:51Z)
• What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review [20.8704278772718]
  We present the results of the first survey of the NLP community. We identify common issues and perspectives on what factors should be considered by paper-reviewer matching systems.
  arXiv  Detail & Related papers  (2022-05-02T16:07:02Z)
• Sentiment Analysis Based on Deep Learning: A Comparative Study [69.09570726777817]
  The study of public opinion can provide us with valuable information. The efficiency and accuracy of sentiment analysis is being hindered by the challenges encountered in natural language processing. This paper reviews the latest studies that have employed deep learning to solve sentiment analysis problems.
  arXiv  Detail & Related papers  (2020-06-05T16:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.