RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements

• URL: http://arxiv.org/abs/2411.15700v1
• Date: Sun, 24 Nov 2024 03:56:43 GMT
• Title: RAMIE: Retrieval-Augmented Multi-task Information Extraction with Large Language Models on Dietary Supplements
• Authors: Zaifu Zhan, Shuang Zhou, Mingchen Li, Rui Zhang,
• Abstract summary: We developed an advanced multi-task large language model (LLM) framework to extract information about dietary supplements (DS) from clinical records. We used four core DS information extraction tasks as our multitasks. With the aid of the RAMIE framework, Llama2-13B achieved an F1 score of 87.39 (3.51% improvement) on the NER task. For the TE task, Llama2-7B scored 79.45 (14.26% improvement), and MedAlpaca-7B achieved the highest F1 score of 93.45 (0.94% improvement) on the
• Score: 12.224815934085154
• License:
• Abstract: \textbf{Objective:} We aimed to develop an advanced multi-task large language model (LLM) framework to extract multiple types of information about dietary supplements (DS) from clinical records. \textbf{Methods:} We used four core DS information extraction tasks - namely, named entity recognition (NER: 2,949 clinical sentences), relation extraction (RE: 4,892 sentences), triple extraction (TE: 2,949 sentences), and usage classification (UC: 2,460 sentences) as our multitasks. We introduced a novel Retrieval-Augmented Multi-task Information Extraction (RAMIE) Framework, including: 1) employed instruction fine-tuning techniques with task-specific prompts, 2) trained LLMs for multiple tasks with improved storage efficiency and lower training costs, and 3) incorporated retrieval augmentation generation (RAG) techniques by retrieving similar examples from the training set. We compared RAMIE's performance to LLMs with instruction fine-tuning alone and conducted an ablation study to assess the contributions of multi-task learning and RAG to improved multitasking performance. \textbf{Results:} With the aid of the RAMIE framework, Llama2-13B achieved an F1 score of 87.39 (3.51\% improvement) on the NER task and demonstrated outstanding performance on the RE task with an F1 score of 93.74 (1.15\% improvement). For the TE task, Llama2-7B scored 79.45 (14.26\% improvement), and MedAlpaca-7B achieved the highest F1 score of 93.45 (0.94\% improvement) on the UC task. The ablation study revealed that while MTL increased efficiency with a slight trade-off in performance, RAG significantly boosted overall accuracy. \textbf{Conclusion:} This study presents a novel RAMIE framework that demonstrates substantial improvements in multi-task information extraction for DS-related data from clinical records. Our framework can potentially be applied to other domains.

Related papers

• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales. We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant [63.28378110792787]
  We introduce LamRA, a versatile framework designed to empower Large Multimodal Models with sophisticated retrieval and reranking capabilities. For retrieval, we adopt a two-stage training strategy comprising language-only pre-training and multimodal instruction tuning. For reranking, we employ joint training for both pointwise and listwise reranking, offering two distinct ways to further boost the retrieval performance.
  arXiv  Detail & Related papers  (2024-12-02T17:10:16Z)
• Instructive Code Retriever: Learn from Large Language Model's Feedback for Code Intelligence Tasks [10.867880635762395]
  We introduce a novel approach named Instructive Code Retriever (ICR) ICR is designed to retrieve examples that enhance model inference across various code intelligence tasks and datasets. We evaluate our model's effectiveness on various tasks, i.e., code summarization, program synthesis, and bug fixing.
  arXiv  Detail & Related papers  (2024-10-15T05:44:00Z)
• RUIE: Retrieval-based Unified Information Extraction using Large Language Model [6.788855739199981]
  Unified information extraction aims to extract structured information from unstructured text. We propose RUIE (Retrieval-based Unified Information Extraction), a framework that leverages in-context learning for efficient task generalization.
  arXiv  Detail & Related papers  (2024-09-18T03:20:04Z)
• Building Math Agents with Multi-Turn Iterative Preference Learning [56.71330214021884]
  This paper studies the complementary direct preference learning approach to further improve model performance. Existing direct preference learning algorithms are originally designed for the single-turn chat task. We introduce a multi-turn direct preference learning framework, tailored for this context.
  arXiv  Detail & Related papers  (2024-09-04T02:41:04Z)
• Less is More: High-value Data Selection for Visual Instruction Tuning [127.38740043393527]
  We propose a high-value data selection approach TIVE, to eliminate redundancy within the visual instruction data and reduce the training cost. Our approach using only about 15% data can achieve comparable average performance to the full-data fine-tuned model across eight benchmarks.
  arXiv  Detail & Related papers  (2024-03-14T16:47:25Z)
• Rethinking and Improving Multi-task Learning for End-to-end Speech Translation [51.713683037303035]
  We investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. We propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation.
  arXiv  Detail & Related papers  (2023-11-07T08:48:46Z)
• BioInstruct: Instruction Tuning of Large Language Models for Biomedical Natural Language Processing [10.698756010878688]
  We created the BioInstruct, comprising 25,005 instructions to instruction-tune large language models (LLMs) The instructions were created by prompting the GPT-4 language model with three-seed samples randomly drawn from an 80 human curated instructions. We evaluated these instruction-tuned LLMs on several BioNLP tasks, which can be grouped into three major categories: question answering(QA), information extraction(IE), and text generation(GEN)
  arXiv  Detail & Related papers  (2023-10-30T19:38:50Z)
• RA-DIT: Retrieval-Augmented Dual Instruction Tuning [90.98423540361946]
  Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option.
  arXiv  Detail & Related papers  (2023-10-02T17:16:26Z)
• Understanding and Improving Information Transfer in Multi-Task Learning [14.43111978531182]
  We study an architecture with a shared module for all tasks and a separate output module for each task. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning.
  arXiv  Detail & Related papers  (2020-05-02T23:43:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.