On the performativity of SDG classifications in large bibliometric databases

• URL: http://arxiv.org/abs/2405.03007v1
• Date: Sun, 5 May 2024 17:28:54 GMT
• Title: On the performativity of SDG classifications in large bibliometric databases
• Authors: Matteo Ottaviani, Stephan Stahlschmidt,
• Abstract summary: Large bibliometric databases have taken up the UN's Sustainable Development Goals in their respective classifications. This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large bibliometric databases, such as Web of Science, Scopus, and OpenAlex, facilitate bibliometric analyses, but are performative, affecting the visibility of scientific outputs and the impact measurement of participating entities. Recently, these databases have taken up the UN's Sustainable Development Goals (SDGs) in their respective classifications, which have been criticised for their diverging nature. This work proposes using the feature of large language models (LLMs) to learn about the "data bias" injected by diverse SDG classifications into bibliometric data by exploring five SDGs. We build a LLM that is fine-tuned in parallel by the diverse SDG classifications inscribed into the databases' SDG classifications. Our results show high sensitivity in model architecture, classified publications, fine-tuning process, and natural language generation. The wide arbitrariness at different levels raises concerns about using LLM in research practice.

Related papers

• Language Models are Graph Learners [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, including Graph Neural Networks (GNNs) and Graph Transformers (GTs) We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models [58.57987316300529]
  Large Language Models (LLMs) are increasingly deployed to handle various natural language processing (NLP) tasks. To evaluate the biases exhibited by LLMs, researchers have recently proposed a variety of datasets. We propose CEB, a Compositional Evaluation Benchmark that covers different types of bias across different social groups and tasks.
  arXiv  Detail & Related papers  (2024-07-02T16:31:37Z)
• BERGEN: A Benchmarking Library for Retrieval-Augmented Generation [26.158785168036662]
  Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. Inconsistent benchmarking poses a major challenge in comparing approaches and understanding the impact of each component in the pipeline. In this work, we study best practices that lay the groundwork for a systematic evaluation of RAG and present BERGEN, an end-to-end library for reproducible research standardizing RAG experiments.
  arXiv  Detail & Related papers  (2024-07-01T09:09:27Z)
• DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph [70.79413606968814]
  We introduce Dynamic Evaluation of LLMs via Adaptive Reasoning Graph Evolvement (DARG) to dynamically extend current benchmarks with controlled complexity and diversity. Specifically, we first extract the reasoning graphs of data points in current benchmarks and then perturb the reasoning graphs to generate novel testing data. Such newly generated test samples can have different levels of complexity while maintaining linguistic diversity similar to the original benchmarks.
  arXiv  Detail & Related papers  (2024-06-25T04:27:53Z)
• Zero-Shot Topic Classification of Column Headers: Leveraging LLMs for Metadata Enrichment [0.0]
  We propose a method to support metadata enrichment using topic annotations generated by three Large Language Models (LLMs): ChatGPT-3.5, GoogleBard, and GoogleGemini. We evaluate the impact of contextual information (i.e., dataset description) on the classification outcomes.
  arXiv  Detail & Related papers  (2024-03-01T10:01:36Z)
• Measuring Distributional Shifts in Text: The Advantage of Language Model-Based Embeddings [11.393822909537796]
  An essential part of monitoring machine learning models in production is measuring input and output data drift. Recent advancements in large language models (LLMs) indicate their effectiveness in capturing semantic relationships. We propose a clustering-based algorithm for measuring distributional shifts in text data by exploiting such embeddings.
  arXiv  Detail & Related papers  (2023-12-04T20:46:48Z)
• GDL-DS: A Benchmark for Geometric Deep Learning under Distribution Shifts [39.21363872039499]
  GDL-DS is a benchmark designed for evaluating the performance of GDL models in scenarios with distribution shifts. Our evaluation datasets cover diverse scientific domains from particle physics and materials science to biochemistry. Overall, our benchmark results in 30 different experiment settings, and evaluates 3 GDL backbones and 11 learning algorithms in each setting.
  arXiv  Detail & Related papers  (2023-10-12T19:27:43Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Graph-based Semi-supervised Learning: A Comprehensive Review [51.26862262550445]
  Semi-supervised learning (SSL) has tremendous value in practice due to its ability to utilize both labeled data and unlabelled data. An important class of SSL methods is to naturally represent data as graphs, which corresponds to graph-based semi-supervised learning (GSSL) methods. GSSL methods have demonstrated their advantages in various domains due to their uniqueness of structure, the universality of applications, and their scalability to large scale data.
  arXiv  Detail & Related papers  (2021-02-26T05:11:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.