A Systematic Review of Reproducibility Research in Natural Language Processing

• URL: http://arxiv.org/abs/2103.07929v1
• Date: Sun, 14 Mar 2021 13:53:05 GMT
• Title: A Systematic Review of Reproducibility Research in Natural Language Processing
• Authors: Anya Belz, Shubham Agarwal, Anastasia Shimorina, Ehud Reiter
• Abstract summary: The past few years have seen an impressive range of new initiatives, events and active research in the area. The field is far from reaching a consensus about how should be defined, measured and addressed.
• Score: 3.0039296468567236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Against the background of what has been termed a reproducibility crisis in science, the NLP field is becoming increasingly interested in, and conscientious about, the reproducibility of its results. The past few years have seen an impressive range of new initiatives, events and active research in the area. However, the field is far from reaching a consensus about how reproducibility should be defined, measured and addressed, with diversity of views currently increasing rather than converging. With this focused contribution, we aim to provide a wide-angle, and as near as possible complete, snapshot of current work on reproducibility in NLP, delineating differences and similarities, and providing pointers to common denominators.

Related papers

• RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluation [73.2390735383842]
  We introduce the first sample-efficient algorithm for LMDPs without any additional structural assumptions. We show how these can be used to derive near-optimal guarantees of an optimistic exploration algorithm. These results can be valuable for a wide range of interactive learning problems beyond LMDPs, and especially, for partially observed environments.
  arXiv  Detail & Related papers  (2024-06-03T14:51:27Z)
• Reproducibility, Replicability, and Repeatability: A survey of reproducible research with a focus on high performance computing [0.0]
  Reproducibility is a fundamental principle in scientific research. Highperformance computing presents unique challenges. This paper provides a comprehensive review of these concerns and potential solutions.
  arXiv  Detail & Related papers  (2024-02-12T09:59:11Z)
• A Comprehensive Survey on Relation Extraction: Recent Advances and New Frontiers [76.51245425667845]
  Relation extraction (RE) involves identifying the relations between entities from underlying content. Deep neural networks have dominated the field of RE and made noticeable progress. This survey is expected to facilitate researchers' collaborative efforts to address the challenges of real-world RE systems.
  arXiv  Detail & Related papers  (2023-06-03T08:39:25Z)
• A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why? [84.46288849132634]
  We propose a systematic framework for analyzing the evolution of research topics in a scientific field using causal discovery and inference techniques. We define three variables to encompass diverse facets of the evolution of research topics within NLP. We utilize a causal discovery algorithm to unveil the causal connections among these variables using observational data.
  arXiv  Detail & Related papers  (2023-05-22T11:08:00Z)
• Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP [84.08476873280644]
  Just 13% of papers had (i) sufficiently low barriers to reproduction, and (ii) enough obtainable information, to be considered for reproduction. As a result, we had to change our coordinated study design from a reproduce approach to a standardise-then-reproduce-twice approach.
  arXiv  Detail & Related papers  (2023-05-02T17:46:12Z)
• Parsing Objects at a Finer Granularity: A Survey [54.72819146263311]
  Fine-grained visual parsing is important in many real-world applications, e.g., agriculture, remote sensing, and space technologies. Predominant research efforts tackle these fine-grained sub-tasks following different paradigms. We conduct an in-depth study of the advanced work from a new perspective of learning the part relationship.
  arXiv  Detail & Related papers  (2022-12-28T04:20:10Z)
• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey [0.0]
  We propose to survey these research works through a new taxonomy based on information theory. We computationally revisit the notions of surprise, novelty and skill learning. Our analysis suggests that novelty and surprise can assist the building of a hierarchy of transferable skills.
  arXiv  Detail & Related papers  (2022-09-19T09:47:43Z)
• Reproducibility in machine learning for medical imaging [3.1390096961027076]
  This chapter intends at being an introduction to for researchers in the field of machine learning for medical imaging. For each of them, we aim at defining it, at describing the requirements to achieve it and at discussing its utility. The chapter ends with a discussion on the benefits of didactic and with a plea for a non-dogmatic approach to this concept and its implementation in research practice.
  arXiv  Detail & Related papers  (2022-09-12T09:00:04Z)
• Vision-Centric BEV Perception: A Survey [92.98068828762833]
  Vision-centric Bird's Eye View (BEV) perception has garnered significant interest from both industry and academia. The rapid advancements in deep learning have led to the proposal of numerous methods for addressing vision-centric BEV perception challenges. This paper compiles and organizes up-to-date knowledge, offering a systematic review and summary of prevalent algorithms.
  arXiv  Detail & Related papers  (2022-08-04T17:53:17Z)
• Quantifying Reproducibility in NLP and ML [0.0]
  Reproducibility has become an intensely debated topic in NLP and ML over recent years. No commonly accepted way of assessing, let alone quantifying it, has so far emerged.
  arXiv  Detail & Related papers  (2021-09-02T21:00:17Z)
• Reproducibility in Evolutionary Computation [0.0]
  We discuss, within the context of EC, the different types of as well as the concepts of artifact and measurement. We identify cultural and technical obstacles to in the EC field. We suggest tools that may help to overcome some of these obstacles.
  arXiv  Detail & Related papers  (2021-02-05T19:06:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.