How broad is that claim? Mapping Generalisation in NLP Research

• URL: http://arxiv.org/abs/2609.14770v1
• Date: Sun, 13 Sep 2026 20:06:36 GMT
• Title: How broad is that claim? Mapping Generalisation in NLP Research
• Abstract summary: We introduce a comprehensive taxonomy of generalisations in the scientific domain, NLPGenX, which labels claims according to their level of generality and framing within the text.<n>We operationalise this taxonomy with an LLM-powered framework, NLPGenA, that automatically classifies sentences from scientific articles into 5 different generalisation classes.<n>We use NLPGens to analyse the use of generalisations in NLP papers across multiple venues and to examine associations with citation counts, hedging, and vague descriptors.
• Score: 3.5572860881327
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Generalisations are common in scientific communication, even though they are semantically ambiguous. An automated method is needed to identify and categorise claims according to their level of generalisation, in order help detect an over-reliance on generalisations and possible misrepresentations of scientific findings. We introduce a comprehensive taxonomy of generalisations in the scientific domain, NLPGenX, which labels claims according to their level of generality and framing within the text. We operationalise this taxonomy with an LLM-powered framework, NLPGenA, that automatically classifies sentences from scientific articles into 5 different generalisation classes. We validate our framework with human annotators and use the framework to construct a large-scale dataset of NLP papers annotated according to generality, with auxiliary labels for hedging and vague descriptors (NLPGens). We use NLPGens to analyse the use of generalisations in NLP papers across multiple venues and subdomains, and to examine associations with citation counts, hedging, and vague descriptors.

Related papers

• GenRubric: Self-Evolving Rubric Generation for Scalable LLM Evaluation [74.44235943118994]
  GenRubric is a self-evolving framework that improves rubric generation from unlabeled queries.<n>We implement this principle through reinforcement learning, combining a cross-rubric comprehensiveness signal with group-level and criterion-level rewards for rubric quality.<n> Experiments on human-annotated rubric benchmarks show that self-evolution improves the agreement between evaluations induced by generated rubrics and those induced by expert-written rubrics.
  arXiv  Detail & Related papers  (2026-08-30T15:39:39Z)
• Term-Centric Hierarchy Induction from Heterogeneous Corpora [39.04226660046721]
  We propose a term-centric framework for inducing hierarchical corpora that scales to massive document collections.<n>Our approach maps documents from diverse sources into a shared representation space using automatic term extraction.<n> Experiments on a novel English and German multi-source benchmark of over one million documents demonstrate that our method improves cross-source coherence and hierarchy quality.
  arXiv  Detail & Related papers  (2026-06-25T12:37:33Z)
• OpenNovelty: An LLM-powered Agentic System for Verifiable Scholarly Novelty Assessment [63.662126457336534]
  OpenNovelty is an agentic system for transparent, evidence-based novelty analysis.<n>It grounds all assessments in retrieved real papers, ensuring verifiable judgments.<n>OpenNovelty aims to empower the research community with a scalable tool that promotes fair, consistent, and evidence-backed peer review.
  arXiv  Detail & Related papers  (2026-01-04T15:48:51Z)
• Context-Aware Hierarchical Taxonomy Generation for Scientific Papers via LLM-Guided Multi-Aspect Clustering [59.54662810933882]
  Existing taxonomy construction methods, leveraging unsupervised clustering or direct prompting of large language models, often lack coherence and granularity.<n>We propose a novel context-aware hierarchical taxonomy generation framework that integrates LLM-guided multi-aspect encoding with dynamic clustering.
  arXiv  Detail & Related papers  (2025-09-23T15:12:58Z)
• Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypes [6.30817290125825]
  We introduce the GUS-Net Framework, comprising the GUS dataset and a multi-label token-level detector for span-level analysis of social bias.<n>The GUS dataset contains 3,739 unique snippets across multiple domains, with over 69,000 token-level annotations.<n>We formulate bias detection as multi-label token-level classification and benchmark both encoder-based models and decoder-based large language models.
  arXiv  Detail & Related papers  (2024-10-10T21:51:22Z)
• Taxonomy Tree Generation from Citation Graph [15.188580557890942]
  HiGTL is a novel end-to-end framework guided by human-provided instructions or preferred topics.<n>We develop a novel taxonomy node verbalization strategy that iteratively generates central concepts for each cluster.<n>Experiments demonstrate that HiGTL effectively produces coherent, high-quality concept.
  arXiv  Detail & Related papers  (2024-10-02T13:02:03Z)
• State-of-the-art generalisation research in NLP: A taxonomy and review [87.1541712509283]
  We present a taxonomy for characterising and understanding generalisation research in NLP. Our taxonomy is based on an extensive literature review of generalisation research. We use our taxonomy to classify over 400 papers that test generalisation.
  arXiv  Detail & Related papers  (2022-10-06T16:53:33Z)
• Understanding Robust Generalization in Learning Regular Languages [85.95124524975202]
  We study robust generalization in the context of using recurrent neural networks to learn regular languages. We propose a compositional strategy to address this. We theoretically prove that the compositional strategy generalizes significantly better than the end-to-end strategy.
  arXiv  Detail & Related papers  (2022-02-20T02:50:09Z)
• Generalizing to Unseen Domains: A Survey on Domain Generalization [59.16754307820612]
  Domain generalization deals with a challenging setting where one or several different but related domain(s) are given. The goal is to learn a model that can generalize to an unseen test domain. This paper presents the first review for recent advances in domain generalization.
  arXiv  Detail & Related papers  (2021-03-02T06:04:11Z)
• LA-HCN: Label-based Attention for Hierarchical Multi-label TextClassification Neural Network [16.12197413284402]
  We propose a Label-based Attention for Hierarchical Mutlti-label Text Classification Neural Network (LA-HCN) LA-HCN is designed to hierarchically extract important information from the text based on the labels from different hierarchy levels. Separate local and global document embeddings are obtained and used to facilitate the respective local and global classifications.
  arXiv  Detail & Related papers  (2020-09-23T06:18:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.