Evaluating the Performance of Large Language Models for SDG Mapping (Technical Report)

• URL: http://arxiv.org/abs/2408.02201v1
• Date: Mon, 5 Aug 2024 03:05:02 GMT
• Title: Evaluating the Performance of Large Language Models for SDG Mapping (Technical Report)
• Authors: Hui Yin, Amir Aryani, Nakul Nambiar,
• Abstract summary: Large language models (LLMs) enable users to protect data privacy by eliminating the need to provide data to third parties. We compare the performance of various language models on the Sustainable Development Goal mapping task. According to the results of this study, LLaMA 2 and Gemma still have significant room for improvement.
• Score: 6.789534723913505
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The use of large language models (LLMs) is expanding rapidly, and open-source versions are becoming available, offering users safer and more adaptable options. These models enable users to protect data privacy by eliminating the need to provide data to third parties and can be customized for specific tasks. In this study, we compare the performance of various language models on the Sustainable Development Goal (SDG) mapping task, using the output of GPT-4o as the baseline. The selected open-source models for comparison include Mixtral, LLaMA 2, LLaMA 3, Gemma, and Qwen2. Additionally, GPT-4o-mini, a more specialized version of GPT-4o, was included to extend the comparison. Given the multi-label nature of the SDG mapping task, we employed metrics such as F1 score, precision, and recall with micro-averaging to evaluate different aspects of the models' performance. These metrics are derived from the confusion matrix to ensure a comprehensive evaluation. We provide a clear observation and analysis of each model's performance by plotting curves based on F1 score, precision, and recall at different thresholds. According to the results of this experiment, LLaMA 2 and Gemma still have significant room for improvement. The other four models do not exhibit particularly large differences in performance. The outputs from all seven models are available on Zenodo: https://doi.org/10.5281/zenodo.12789375.

Related papers

• Foundational GPT Model for MEG [3.524869467682149]
  We propose two classes of deep learning foundational models that can be trained using forecasting of unlabelled brain signals. First, we consider a modified Wavenet; and second, we consider a modified Transformer-based (GPT2) model. We compare the performance of these deep learning models with standard linear autoregressive (AR) modelling on MEG data.
  arXiv  Detail & Related papers  (2024-04-14T13:48:24Z)
• MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models [70.92847554971065]
  We introduce MT-Eval, a comprehensive benchmark designed to evaluate multi-turn conversational abilities. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. Our evaluation of 11 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks.
  arXiv  Detail & Related papers  (2024-01-30T04:50:28Z)
• PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging [8.043625583479598]
  Multimodal large language models (MLLMs) represent an evolutionary expansion in the capabilities of traditional large language models. Recent works investigate the adaptation of MLLMs as a universal solution to address medical multi-modal problems as a generative task. We propose a parameter efficient framework for fine-tuning MLLMs, specifically validated on medical visual question answering (Med-VQA) and medical report generation (MRG) tasks.
  arXiv  Detail & Related papers  (2024-01-05T13:22:12Z)
• Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases [98.35348038111508]
  This paper presents an in-depth comparative study of two pioneering models: Google's Gemini and OpenAI's GPT-4V(ision) The core of our analysis delves into the distinct visual comprehension abilities of each model. Our findings illuminate the unique strengths and niches of both models.
  arXiv  Detail & Related papers  (2023-12-22T18:59:58Z)
• Large language models for aspect-based sentiment analysis [0.0]
  We assess the performance of GPT-4 and GPT-3.5 in zero shot, few shot and fine-tuned settings. Fine-tuned GPT-3.5 achieves a state-of-the-art F1 score of 83.8 on the joint aspect term extraction and polarity classification task.
  arXiv  Detail & Related papers  (2023-10-27T10:03:21Z)
• Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency [127.97467912117652]
  Large language models (LLMs) have exhibited remarkable ability in code generation. However, generating the correct solution in a single attempt still remains a challenge. We propose the Multi-Perspective Self-Consistency (MPSC) framework incorporating both inter- and intra-consistency.
  arXiv  Detail & Related papers  (2023-09-29T14:23:26Z)
• Text Alignment Is An Efficient Unified Model for Massive NLP Tasks [24.069447197357164]
  Next-word prediction is often not an efficient formulation for many NLP tasks. We propose text alignment as an efficient unified model for a wide range of crucial tasks. Our model delivers on par or even superior performance with much smaller model sizes.
  arXiv  Detail & Related papers  (2023-07-06T02:28:31Z)
• How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources [117.6496550359768]
  This work explores recent advances in instruction-tuning language models on a range of open instruction-following datasets. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets. We evaluate them on their factual knowledge, reasoning, multilinguality, coding, and open-ended instruction following abilities.
  arXiv  Detail & Related papers  (2023-06-07T19:59:23Z)
• Explanation-based Finetuning Makes Models More Robust to Spurious Cues [21.327036110196637]
  Large Language Models (LLMs) are so powerful that they sometimes learn correlations between labels and features that are irrelevant to the task. We propose explanation-based finetuning as a general approach to mitigate LLMs' reliance on spurious correlations. We finetune the model to additionally generate a free-text explanation supporting its answer.
  arXiv  Detail & Related papers  (2023-05-08T18:53:45Z)
• Exploring the Trade-Offs: Unified Large Language Models vs Local Fine-Tuned Models for Highly-Specific Radiology NLI Task [49.50140712943701]
  We evaluate the performance of ChatGPT/GPT-4 on a radiology NLI task and compare it to other models fine-tuned specifically on task-related data samples. We also conduct a comprehensive investigation on ChatGPT/GPT-4's reasoning ability by introducing varying levels of inference difficulty.
  arXiv  Detail & Related papers  (2023-04-18T17:21:48Z)
• Efficient Large Scale Language Modeling with Mixtures of Experts [61.45159383372181]
  Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings.
  arXiv  Detail & Related papers  (2021-12-20T17:05:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.