Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentation

• URL: http://arxiv.org/abs/2408.03127v1
• Date: Tue, 6 Aug 2024 11:59:09 GMT
• Title: Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentation
• Authors: Artur Guimarães, Bruno Martins, João Magalhães,
• Abstract summary: We develop a prompt for the NLI4CT task, and fine-tune a quantized version of the model using an augmented version of the training dataset. The experimental results show that this approach can produce notable results in terms of the macro F1-score.
• Score: 6.655410984703003
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This paper describes our approach to the SemEval-2024 safe biomedical Natural Language Inference for Clinical Trials (NLI4CT) task, which concerns classifying statements about Clinical Trial Reports (CTRs). We explored the capabilities of Mistral-7B, a generalist open-source Large Language Model (LLM). We developed a prompt for the NLI4CT task, and fine-tuned a quantized version of the model using an augmented version of the training dataset. The experimental results show that this approach can produce notable results in terms of the macro F1-score, while having limitations in terms of faithfulness and consistency. All the developed code is publicly available on a GitHub repository

Related papers

• Towards Evaluating and Building Versatile Large Language Models for Medicine [57.49547766838095]
  We present MedS-Bench, a benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks.
  arXiv  Detail & Related papers  (2024-08-22T17:01:34Z)
• SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature [80.49349719239584]
  We present SciRIFF (Scientific Resource for Instruction-Following and Finetuning), a dataset of 137K instruction-following demonstrations for 54 tasks. SciRIFF is the first dataset focused on extracting and synthesizing information from research literature across a wide range of scientific fields.
  arXiv  Detail & Related papers  (2024-06-10T21:22:08Z)
• Universal and Extensible Language-Vision Models for Organ Segmentation and Tumor Detection from Abdominal Computed Tomography [50.08496922659307]
  We propose a universal framework enabling a single model, termed Universal Model, to deal with multiple public datasets and adapt to new classes. Firstly, we introduce a novel language-driven parameter generator that leverages language embeddings from large language models. Secondly, the conventional output layers are replaced with lightweight, class-specific heads, allowing Universal Model to simultaneously segment 25 organs and six types of tumors.
  arXiv  Detail & Related papers  (2024-05-28T16:55:15Z)
• SEME at SemEval-2024 Task 2: Comparing Masked and Generative Language Models on Natural Language Inference for Clinical Trials [0.9012198585960441]
  This paper describes our submission to Task 2 of SemEval-2024: Safe Biomedical Natural Language Inference for Clinical Trials. The Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) consists of a Textual Entailment task focused on the evaluation of the consistency and faithfulness of Natural Language Inference (NLI) models applied to Clinical Trial Reports (CTR)
  arXiv  Detail & Related papers  (2024-04-05T09:18:50Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• The All-Seeing Project V2: Towards General Relation Comprehension of the Open World [58.40101895719467]
  We present the All-Seeing Project V2, a new model and dataset designed for understanding object relations in images. We propose the All-Seeing Model V2 that integrates the formulation of text generation, object localization, and relation comprehension into a relation conversation task. Our model excels not only in perceiving and recognizing all objects within the image but also in grasping the intricate relation graph between them.
  arXiv  Detail & Related papers  (2024-02-29T18:59:17Z)
• Exploring the Effectiveness of Instruction Tuning in Biomedical Language Processing [19.41164870575055]
  This study investigates the potential of instruction tuning for biomedical language processing. We present a comprehensive, instruction-based model trained on a dataset that consists of approximately $200,000$ instruction-focused samples.
  arXiv  Detail & Related papers  (2023-12-31T20:02:10Z)
• Exploring Prompting Large Language Models as Explainable Metrics [0.0]
  We propose a zero-shot prompt-based strategy for explainable evaluation of the summarization task using Large Language Models (LLMs) The conducted experiments demonstrate the promising potential of LLMs as evaluation metrics in Natural Language Processing (NLP) The performance of our best provided prompts achieved a Kendall correlation of 0.477 with human evaluations in the text summarization task on the test data.
  arXiv  Detail & Related papers  (2023-11-20T06:06:22Z)
• PULSAR at MEDIQA-Sum 2023: Large Language Models Augmented by Synthetic Dialogue Convert Patient Dialogues to Medical Records [23.25763256861649]
  This paper describes PULSAR, our system submission at the ImageClef 2023 MediQA-Sum task on summarising patient-doctor dialogues into clinical records. The proposed framework relies on domain-specific pre-training, to produce a specialised language model which is trained on task-specific natural data. We find limited evidence towards the efficacy of domain-specific pre-training and data augmentation, while scaling up the language model yields the best performance gains.
  arXiv  Detail & Related papers  (2023-07-05T03:31:12Z)
• NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports [3.0468533447146244]
  We present a novel resource to advance research on NLI for reasoning on clinical trial reports. We provide NLI4CT, a corpus of 2400 statements and CTRs, annotated for these tasks. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs.
  arXiv  Detail & Related papers  (2023-05-05T15:03:01Z)
• Development and validation of a natural language processing algorithm to pseudonymize documents in the context of a clinical data warehouse [53.797797404164946]
  The study highlights the difficulties faced in sharing tools and resources in this domain. We annotated a corpus of clinical documents according to 12 types of identifying entities. We build a hybrid system, merging the results of a deep learning model as well as manual rules.
  arXiv  Detail & Related papers  (2023-03-23T17:17:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.