State-of-the-art generalisation research in NLP: A taxonomy and review
- URL: http://arxiv.org/abs/2210.03050v4
- Date: Fri, 12 Jan 2024 13:11:20 GMT
- Title: State-of-the-art generalisation research in NLP: A taxonomy and review
- Authors: Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai
Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi
Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar
Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita
Frieske, Ryan Cotterell, Zhijing Jin
- Abstract summary: We present a taxonomy for characterising and understanding generalisation research in NLP.
Our taxonomy is based on an extensive literature review of generalisation research.
We use our taxonomy to classify over 400 papers that test generalisation.
- Score: 87.1541712509283
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The ability to generalise well is one of the primary desiderata of natural
language processing (NLP). Yet, what 'good generalisation' entails and how it
should be evaluated is not well understood, nor are there any evaluation
standards for generalisation. In this paper, we lay the groundwork to address
both of these issues. We present a taxonomy for characterising and
understanding generalisation research in NLP. Our taxonomy is based on an
extensive literature review of generalisation research, and contains five axes
along which studies can differ: their main motivation, the type of
generalisation they investigate, the type of data shift they consider, the
source of this data shift, and the locus of the shift within the modelling
pipeline. We use our taxonomy to classify over 400 papers that test
generalisation, for a total of more than 600 individual experiments.
Considering the results of this review, we present an in-depth analysis that
maps out the current state of generalisation research in NLP, and we make
recommendations for which areas might deserve attention in the future. Along
with this paper, we release a webpage where the results of our review can be
dynamically explored, and which we intend to update as new NLP generalisation
studies are published. With this work, we aim to take steps towards making
state-of-the-art generalisation testing the new status quo in NLP.
Related papers
- RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance [0.8089605035945486]
We propose RelevAI-Reviewer, an automatic system that conceptualizes the task of survey paper review as a classification problem.
We introduce a novel dataset comprised of 25,164 instances. Each instance contains one prompt and four candidate papers, each varying in relevance to the prompt.
We develop a machine learning (ML) model capable of determining the relevance of each paper and identifying the most pertinent one.
arXiv Detail & Related papers (2024-06-13T06:42:32Z) - A Literature Review of Literature Reviews in Pattern Analysis and Machine Intelligence [58.6354685593418]
This paper proposes several article-level, field-normalized, and large language model-empowered bibliometric indicators to evaluate reviews.
The newly emerging AI-generated literature reviews are also appraised.
This work offers insights into the current challenges of literature reviews and envisions future directions for their development.
arXiv Detail & Related papers (2024-02-20T11:28:50Z) - This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language
Models [12.214260053244871]
We analyse the body of work that uses prompts and templates to assess bias in language models.
We draw on a measurement modelling framework to create a taxonomy of attributes that capture what a bias test aims to measure.
Our analysis illuminates the scope of possible bias types the field is able to measure, and reveals types that are as yet under-researched.
arXiv Detail & Related papers (2023-05-22T06:28:48Z) - Investigating Fairness Disparities in Peer Review: A Language Model
Enhanced Approach [77.61131357420201]
We conduct a thorough and rigorous study on fairness disparities in peer review with the help of large language models (LMs)
We collect, assemble, and maintain a comprehensive relational database for the International Conference on Learning Representations (ICLR) conference from 2017 to date.
We postulate and study fairness disparities on multiple protective attributes of interest, including author gender, geography, author, and institutional prestige.
arXiv Detail & Related papers (2022-11-07T16:19:42Z) - Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the
Research Manifold [88.83876819883653]
We show through a manual classification of recent NLP research papers that this is indeed the case.
We observe that NLP research often goes beyond the square one setup, focusing not only on accuracy, but also on fairness or interpretability, but typically only along a single dimension.
arXiv Detail & Related papers (2022-06-20T13:04:23Z) - Towards Out-Of-Distribution Generalization: A Survey [46.329995334444156]
Out-of-Distribution generalization is an emerging topic of machine learning research.
This paper represents the first comprehensive, systematic review of OOD generalization.
arXiv Detail & Related papers (2021-08-31T05:28:42Z) - Domain Generalization: A Survey [146.68420112164577]
Domain generalization (DG) aims to achieve OOD generalization by only using source domain data for model learning.
For the first time, a comprehensive literature review is provided to summarize the ten-year development in DG.
arXiv Detail & Related papers (2021-03-03T16:12:22Z) - Automatic generation of reviews of scientific papers [1.1999555634662633]
We present a method for the automatic generation of a review paper corresponding to a user-defined query.
The first part identifies key papers in the area by their bibliometric parameters, such as a graph of co-citations.
The second stage uses a BERT based architecture that we train on existing reviews for extractive summarization of these key papers.
arXiv Detail & Related papers (2020-10-08T17:47:07Z) - A Survey on Text Classification: From Shallow to Deep Learning [83.47804123133719]
The last decade has seen a surge of research in this area due to the unprecedented success of deep learning.
This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021.
We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification.
arXiv Detail & Related papers (2020-08-02T00:09:03Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.