Zero-shot data citation function classification using transformer-based large language models (LLMs)

• URL: http://arxiv.org/abs/2511.02936v1
• Date: Tue, 04 Nov 2025 19:33:30 GMT
• Title: Zero-shot data citation function classification using transformer-based large language models (LLMs)
• Authors: Neil Byers, Ali Zaidi, Valerie Skye, Chris Beecroft, Kjiersten Fagnan,
• Abstract summary: We apply an open-source large language model to generate structured data use case labels for publications known to incorporate specific genomic datasets.<n>Our results demonstrate that the stock model can achieve an F1 score of.674 on a zero-shot data citation classification task with no previously defined categories.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Efforts have increased in recent years to identify associations between specific datasets and the scientific literature that incorporates them. Knowing that a given publication cites a given dataset, the next logical step is to explore how or why that data was used. Advances in recent years with pretrained, transformer-based large language models (LLMs) offer potential means for scaling the description of data use cases in the published literature. This avoids expensive manual labeling and the development of training datasets for classical machine-learning (ML) systems. In this work we apply an open-source LLM, Llama 3.1-405B, to generate structured data use case labels for publications known to incorporate specific genomic datasets. We also introduce a novel evaluation framework for determining the efficacy of our methods. Our results demonstrate that the stock model can achieve an F1 score of .674 on a zero-shot data citation classification task with no previously defined categories. While promising, our results are qualified by barriers related to data availability, prompt overfitting, computational infrastructure, and the expense required to conduct responsible performance evaluation.

Related papers

• Large Language Models and Synthetic Data for Monitoring Dataset Mentions in Research Papers [0.0]
  This paper presents a machine learning framework that automates dataset mention detection across research domains.<n>We employ zero-shot extraction from research papers, an LLM-as-a-Judge for quality assessment, and a reasoning agent for refinement to generate a weakly supervised synthetic dataset.<n>At inference, a ModernBERT-based classifier efficiently filters dataset mentions, reducing computational overhead while maintaining high recall.
  arXiv  Detail & Related papers  (2025-02-14T16:16:02Z)
• The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection [23.378592856800168]
  Large Language Models (LLMs) can be used to automate the annotation process.<n>This study investigates whether LLMs are viable for annotating the complex task of media bias detection.<n>We create annolexical, the first large-scale dataset for media bias classification.
  arXiv  Detail & Related papers  (2024-11-17T14:14:36Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• A Simple Baseline for Predicting Events with Auto-Regressive Tabular Transformers [70.20477771578824]
  Existing approaches to event prediction include time-aware positional embeddings, learned row and field encodings, and oversampling methods for addressing class imbalance. We propose a simple but flexible baseline using standard autoregressive LLM-style transformers with elementary positional embeddings and a causal language modeling objective. Our baseline outperforms existing approaches across popular datasets and can be employed for various use-cases.
  arXiv  Detail & Related papers  (2024-10-14T15:59:16Z)
• Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisions [62.12545440385489]
  Large language models (LLMs) have brought substantial advancements in text generation, but their potential for enhancing classification tasks remains underexplored. We propose a framework for thoroughly investigating fine-tuning LLMs for classification, including both generation- and encoding-based approaches. We instantiate this framework in edit intent classification (EIC), a challenging and underexplored classification task.
  arXiv  Detail & Related papers  (2024-10-02T20:48:28Z)
• LLM-Select: Feature Selection with Large Language Models [64.5099482021597]
  Large language models (LLMs) are capable of selecting the most predictive features, with performance rivaling the standard tools of data science.<n>Our findings suggest that LLMs may be useful not only for selecting the best features for training but also for deciding which features to collect in the first place.
  arXiv  Detail & Related papers  (2024-07-02T22:23:40Z)
• A Context-Aware Approach for Enhancing Data Imputation with Pre-trained Language Models [0.18416014644193068]
  CRILM uses pre-trained language models to create contextually relevant descriptors for missing values.<n>Our evaluations demonstrate CRILM's superior performance and robustness across MCAR, MAR, and challenging MNAR scenarios.
  arXiv  Detail & Related papers  (2024-05-28T00:08:29Z)
• DataAgent: Evaluating Large Language Models' Ability to Answer Zero-Shot, Natural Language Queries [0.0]
  We evaluate OpenAI's GPT-3.5 as a "Language Data Scientist" (LDS) The model was tested on a diverse set of benchmark datasets to evaluate its performance across multiple standards.
  arXiv  Detail & Related papers  (2024-03-29T22:59:34Z)
• Large Language Models for Data Annotation and Synthesis: A Survey [49.8318827245266]
  This survey focuses on the utility of Large Language Models for data annotation and synthesis.<n>It includes an in-depth taxonomy of data types that LLMs can annotate, a review of learning strategies for models utilizing LLM-generated annotations, and a detailed discussion of the primary challenges and limitations associated with using LLMs for data annotation and synthesis.
  arXiv  Detail & Related papers  (2024-02-21T00:44:04Z)
• Utilising a Large Language Model to Annotate Subject Metadata: A Case Study in an Australian National Research Data Catalogue [18.325675189960833]
  In support of open and reproducible research, there has been a rapidly increasing number of datasets made available for research. As the availability of datasets increases, it becomes more important to have quality metadata for discovering and reusing them. This paper proposes to leverage large language models (LLMs) for cost-effective annotation of subject metadata through the LLM-based in-context learning.
  arXiv  Detail & Related papers  (2023-10-17T14:52:33Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)
• Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
  We investigate three schemes to improve the model generalization ability for few-shot settings. We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data. We create new state-of-the-art results on both few-shot and training-free settings.
  arXiv  Detail & Related papers  (2020-12-29T23:43:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.