NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports

• URL: http://arxiv.org/abs/2305.03598v3
• Date: Sat, 28 Oct 2023 09:56:49 GMT
• Title: NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
• Authors: Ma\"el Jullien, Marco Valentino, Hannah Frost, Paul O'Regan, Donal Landers, and Andr\'e Freitas
• Abstract summary: We present a novel resource to advance research on NLI for reasoning on clinical trial reports. We provide NLI4CT, a corpus of 2400 statements and CTRs, annotated for these tasks. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs.
• Score: 3.0468533447146244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How can we interpret and retrieve medical evidence to support clinical decisions? Clinical trial reports (CTR) amassed over the years contain indispensable information for the development of personalized medicine. However, it is practically infeasible to manually inspect over 400,000+ clinical trial reports in order to find the best evidence for experimental treatments. Natural Language Inference (NLI) offers a potential solution to this problem, by allowing the scalable computation of textual entailment. However, existing NLI models perform poorly on biomedical corpora, and previously published datasets fail to capture the full complexity of inference over CTRs. In this work, we present a novel resource to advance research on NLI for reasoning on CTRs. The resource includes two main tasks. Firstly, to determine the inference relation between a natural language statement, and a CTR. Secondly, to retrieve supporting facts to justify the predicted relation. We provide NLI4CT, a corpus of 2400 statements and CTRs, annotated for these tasks. Baselines on this corpus expose the limitations of existing NLI models, with 6 state-of-the-art NLI models achieving a maximum F1 score of 0.627. To the best of our knowledge, we are the first to design a task that covers the interpretation of full CTRs. To encourage further work on this challenging dataset, we make the corpus, competition leaderboard, website and code to replicate the baseline experiments available at: https://github.com/ai-systems/nli4ct

Related papers

• Lisbon Computational Linguists at SemEval-2024 Task 2: Using A Mistral 7B Model and Data Augmentation [6.655410984703003]
  We develop a prompt for the NLI4CT task, and fine-tune a quantized version of the model using an augmented version of the training dataset. The experimental results show that this approach can produce notable results in terms of the macro F1-score.
  arXiv  Detail & Related papers  (2024-08-06T11:59:09Z)
• A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus [71.77214818319054]
  Natural language inference is a proxy for natural language understanding. There is no publicly available NLI corpus for the Romanian language. We introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs.
  arXiv  Detail & Related papers  (2024-05-20T08:41:15Z)
• MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference [65.37685198688538]
  This paper presents MSciNLI, a dataset containing 132,320 sentence pairs extracted from five new scientific domains. We establish strong baselines on MSciNLI by fine-tuning Pre-trained Language Models (PLMs) and prompting Large Language Models (LLMs) We show that domain shift degrades the performance of scientific NLI models which demonstrates the diverse characteristics of different domains in our dataset.
  arXiv  Detail & Related papers  (2024-04-11T18:12:12Z)
• SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials [13.59675117792588]
  We present SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for ClinicalTrials. Our contributions include the refined NLI4CT-P dataset (i.e., Natural Language Inference for Clinical Trials - Perturbed) A total of 106 participants registered for the task contributing to over 1200 individual submissions and 25 system overview papers. This initiative aims to advance the robustness and applicability of NLI models in healthcare, ensuring safer and more dependable AI assistance in clinical decision-making.
  arXiv  Detail & Related papers  (2024-04-07T13:58:41Z)
• Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models [48.07083163501746]
  Clinical natural language processing requires methods that can address domain-specific challenges. We propose an innovative, resource-efficient approach, ClinGen, which infuses knowledge into the process. Our empirical study across 7 clinical NLP tasks and 16 datasets reveals that ClinGen consistently enhances performance across various tasks.
  arXiv  Detail & Related papers  (2023-11-01T04:37:28Z)
• THiFLY Research at SemEval-2023 Task 7: A Multi-granularity System for CTR-based Textual Entailment and Evidence Retrieval [13.30918296659228]
  The NLI4CT task aims to entail hypotheses based on Clinical Trial Reports (CTRs) and retrieve the corresponding evidence supporting the justification. We present a multi-granularity system for CTR-based textual entailment and evidence retrieval. We enhance the numerical inference capability of the system by leveraging a T5-based model, SciFive, which is pre-trained on the medical corpus.
  arXiv  Detail & Related papers  (2023-06-02T03:09:31Z)
• Development and validation of a natural language processing algorithm to pseudonymize documents in the context of a clinical data warehouse [53.797797404164946]
  The study highlights the difficulties faced in sharing tools and resources in this domain. We annotated a corpus of clinical documents according to 12 types of identifying entities. We build a hybrid system, merging the results of a deep learning model as well as manual rules.
  arXiv  Detail & Related papers  (2023-03-23T17:17:46Z)
• Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusters [35.103851212995046]
  Natural Language Inference (NLI) has been extensively studied by the NLP community as a framework for estimating the semantic relation between sentence pairs. We explore the direct zero-shot applicability of NLI models to real applications, beyond the sentence-pair setting they were trained on. We develop new aggregation methods to allow operating over full documents, reaching state-of-the-art performance on the ContractNLI dataset.
  arXiv  Detail & Related papers  (2022-04-15T12:56:39Z)
• DocNLI: A Large-scale Dataset for Document-level Natural Language Inference [55.868482696821815]
  Natural language inference (NLI) is formulated as a unified framework for solving various NLP problems. This work presents DocNLI -- a newly-constructed large-scale dataset for document-level NLI.
  arXiv  Detail & Related papers  (2021-06-17T13:02:26Z)
• Text Mining to Identify and Extract Novel Disease Treatments From Unstructured Datasets [56.38623317907416]
  We use Google Cloud to transcribe podcast episodes of an NPR radio show. We then build a pipeline for systematically pre-processing the text. Our model successfully identified that Omeprazole can help treat heartburn.
  arXiv  Detail & Related papers  (2020-10-22T19:52:49Z)
• Evidence Inference 2.0: More Data, Better Models [22.53884716373888]
  The Evidence Inference dataset was recently released to facilitate research toward this end. This paper collects additional annotations to expand the Evidence Inference dataset by 25%. The updated corpus, documentation, and code for new baselines and evaluations are available at http://evidence-inference.ebm-nlp.com/.
  arXiv  Detail & Related papers  (2020-05-08T17:16:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.