Zero-Shot Topic Classification of Column Headers: Leveraging LLMs for Metadata Enrichment

• URL: http://arxiv.org/abs/2403.00884v3
• Date: Fri, 6 Sep 2024 14:49:21 GMT
• Title: Zero-Shot Topic Classification of Column Headers: Leveraging LLMs for Metadata Enrichment
• Authors: Margherita Martorana, Tobias Kuhn, Lise Stork, Jacco van Ossenbruggen,
• Abstract summary: We propose a method to support metadata enrichment using topic annotations generated by three Large Language Models (LLMs): ChatGPT-3.5, GoogleBard, and GoogleGemini. We evaluate the impact of contextual information (i.e., dataset description) on the classification outcomes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional dataset retrieval systems rely on metadata for indexing, rather than on the underlying data values. However, high-quality metadata creation and enrichment often require manual annotations, which is a labour-intensive and challenging process to automate. In this study, we propose a method to support metadata enrichment using topic annotations generated by three Large Language Models (LLMs): ChatGPT-3.5, GoogleBard, and GoogleGemini. Our analysis focuses on classifying column headers based on domain-specific topics from the Consortium of European Social Science Data Archives (CESSDA), a Linked Data controlled vocabulary. Our approach operates in a zero-shot setting, integrating the controlled topic vocabulary directly within the input prompt. This integration serves as a Large Context Windows approach, with the aim of improving the results of the topic classification task. We evaluated the performance of the LLMs in terms of internal consistency, inter-machine alignment, and agreement with human classification. Additionally, we investigate the impact of contextual information (i.e., dataset description) on the classification outcomes. Our findings suggest that ChatGPT and GoogleGemini outperform GoogleBard in terms of internal consistency as well as LLM-human-agreement. Interestingly, we found that contextual information had no significant impact on LLM performance. This work proposes a novel approach that leverages LLMs for topic classification of column headers using a controlled vocabulary, presenting a practical application of LLMs and Large Context Windows within the Semantic Web domain. This approach has the potential to facilitate automated metadata enrichment, thereby enhancing dataset retrieval and the Findability, Accessibility, Interoperability, and Reusability (FAIR) of research data on the Web.

Related papers

• Evaluating LLM Prompts for Data Augmentation in Multi-label Classification of Ecological Texts [1.565361244756411]
  Large language models (LLMs) play a crucial role in natural language processing (NLP) tasks. This study applied prompt-based data augmentation to detect mentions of green practices in Russian social media.
  arXiv  Detail & Related papers  (2024-11-22T12:37:41Z)
• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• Text Clustering as Classification with LLMs [6.030435811868953]
  This study presents a novel framework for text clustering that effectively leverages the in-context learning capacity of Large Language Models (LLMs) Instead of fine-tuning embedders, we propose to transform the text clustering into a classification task via LLM. Our framework has been experimentally proven to achieve comparable or superior performance to state-of-the-art clustering methods.
  arXiv  Detail & Related papers  (2024-09-30T16:57:34Z)
• LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation [67.24113079928668]
  We present LexMatcher, a method for data curation driven by the coverage of senses found in bilingual dictionaries. Our approach outperforms the established baselines on the WMT2022 test sets.
  arXiv  Detail & Related papers  (2024-06-03T15:30:36Z)
• Text-Video Retrieval with Global-Local Semantic Consistent Learning [122.15339128463715]
  We propose a simple yet effective method, Global-Local Semantic Consistent Learning (GLSCL) GLSCL capitalizes on latent shared semantics across modalities for text-video retrieval. Our method achieves comparable performance with SOTA as well as being nearly 220 times faster in terms of computational cost.
  arXiv  Detail & Related papers  (2024-05-21T11:59:36Z)
• Utilising a Large Language Model to Annotate Subject Metadata: A Case Study in an Australian National Research Data Catalogue [18.325675189960833]
  In support of open and reproducible research, there has been a rapidly increasing number of datasets made available for research. As the availability of datasets increases, it becomes more important to have quality metadata for discovering and reusing them. This paper proposes to leverage large language models (LLMs) for cost-effective annotation of subject metadata through the LLM-based in-context learning.
  arXiv  Detail & Related papers  (2023-10-17T14:52:33Z)
• MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data. For quality, we resort to GPT-4 to generate high-quality data with each given data type.
  arXiv  Detail & Related papers  (2023-08-25T01:41:04Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method [35.181659789684545]
  Automatic summarization generates concise summaries that contain key ideas of source documents. References from CNN/DailyMail and BBC XSum are noisy, mainly in terms of factual hallucination and information redundancy. We propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L.
  arXiv  Detail & Related papers  (2023-05-22T18:54:35Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.