Instruction Finetuning for Leaderboard Generation from Empirical AI Research

• URL: http://arxiv.org/abs/2408.10141v1
• Date: Mon, 19 Aug 2024 16:41:07 GMT
• Title: Instruction Finetuning for Leaderboard Generation from Empirical AI Research
• Authors: Salomon Kabongo, Jennifer D'Souza,
• Abstract summary: This study demonstrates the application of instruction finetuning of Large Language Models (LLMs) to automate the generation of AI research leaderboards. It aims to streamline the dissemination of advancements in AI research by transitioning from traditional, manual community curation.
• Score: 0.16114012813668935
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study demonstrates the application of instruction finetuning of pretrained Large Language Models (LLMs) to automate the generation of AI research leaderboards, extracting (Task, Dataset, Metric, Score) quadruples from articles. It aims to streamline the dissemination of advancements in AI research by transitioning from traditional, manual community curation, or otherwise taxonomy-constrained natural language inference (NLI) models, to an automated, generative LLM-based approach. Utilizing the FLAN-T5 model, this research enhances LLMs' adaptability and reliability in information extraction, offering a novel method for structured knowledge representation.

Related papers

• SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models [54.78329741186446]
  We propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. Experiments across both in-domain and out-of-domain benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
  arXiv  Detail & Related papers  (2024-08-28T06:33:03Z)
• Automating Knowledge Discovery from Scientific Literature via LLMs: A Dual-Agent Approach with Progressive Ontology Prompting [59.97247234955861]
  We introduce a novel framework based on large language models (LLMs) that combines a progressive prompting algorithm with a dual-agent system, named LLM-Duo. Our method identifies 2,421 interventions from 64,177 research articles in the speech-language therapy domain.
  arXiv  Detail & Related papers  (2024-08-20T16:42:23Z)
• Effective Context Selection in LLM-based Leaderboard Generation: An Empirical Study [0.3072340427031969]
  This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating AI research leaderboards. We introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy.
  arXiv  Detail & Related papers  (2024-06-06T06:05:39Z)
• Exploring the Latest LLMs for Leaderboard Extraction [0.3072340427031969]
  This paper investigates the efficacy of different LLMs-ralMist 7B, Llama GPT-4-Turbo and GPT-4.o in extracting leaderboard information from empirical AI research articles. Our study evaluates the performance of these models in generating (Task, Metric, Score) quadruples from research papers.
  arXiv  Detail & Related papers  (2024-06-06T05:54:45Z)
• LLMs for XAI: Future Directions for Explaining Explanations [50.87311607612179]
  We focus on refining explanations computed using existing XAI algorithms. Initial experiments and user study suggest that LLMs offer a promising way to enhance the interpretability and usability of XAI.
  arXiv  Detail & Related papers  (2024-05-09T19:17:47Z)
• Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning [0.9110413356918055]
  This research pioneers the use of fine-tuned Large Language Models (LLMs) to automate Systematic Literature Reviews ( SLRs) Our study employed the latest fine-tuning methodologies together with open-sourced LLMs, and demonstrated a practical and efficient approach to automating the final execution stages of an SLR process. The results maintained high fidelity in factual accuracy in LLM responses, and were validated through the replication of an existing PRISMA-conforming SLR.
  arXiv  Detail & Related papers  (2024-04-08T00:08:29Z)
• Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models [3.0566617373924325]
  Sentence embedding is a fundamental task within the realm of Natural Language Processing, finding extensive application in search engines, expert systems, and question-and-answer platforms. With the continuous evolution of large language models such as LLaMA and Mistral, research on sentence embedding has recently achieved notable breakthroughs. We propose two innovative prompt engineering techniques capable of further enhancing the expressive power of PLMs' raw embeddings.
  arXiv  Detail & Related papers  (2024-04-05T07:07:15Z)
• Large Language Models for Scientific Information Extraction: An Empirical Study for Virology [0.0]
  We champion the use of structured and semantic content representation of discourse-based scholarly communication. Inspired by tools like Wikipedia infoboxes or structured Amazon product descriptions, we develop an automated approach to produce structured scholarly contribution summaries. Our results show that finetuned FLAN-T5 with 1000x fewer parameters than the state-of-the-art GPT-davinci is competitive for the task.
  arXiv  Detail & Related papers  (2024-01-18T15:04:55Z)
• Instruction Tuning for Large Language Models: A Survey [52.86322823501338]
  This paper surveys research works in the quickly advancing field of instruction tuning (IT) In this paper, unless specified otherwise, instruction tuning (IT) will be equivalent to supervised fine-tuning (SFT)
  arXiv  Detail & Related papers  (2023-08-21T15:35:16Z)
• A Survey on Large Language Models for Recommendation [77.91673633328148]
  Large Language Models (LLMs) have emerged as powerful tools in the field of Natural Language Processing (NLP) This survey presents a taxonomy that categorizes these models into two major paradigms, respectively Discriminative LLM for Recommendation (DLLM4Rec) and Generative LLM for Recommendation (GLLM4Rec)
  arXiv  Detail & Related papers  (2023-05-31T13:51:26Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.