Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting

• URL: http://arxiv.org/abs/2402.12801v2
• Date: Tue, 08 Oct 2024 11:59:07 GMT
• Title: Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting
• Authors: Marco Naguib, Xavier Tannier, Aurélie Névéol,
• Abstract summary: Large language models (LLMs) have become the preferred solution for many natural language processing tasks. This study aims to evaluate generative LLMs, employed through prompt engineering, for few-shot clinical NER. We compare 13 auto-regressive models using prompting and 16 masked models using fine-tuning on 14 NER datasets covering English, French and Spanish. While prompt-based auto-regressive models achieve competitive F1 for general NER, they are outperformed within the clinical domain by lighter biLSTM-CRF taggers based on masked models.
• Score: 4.832840259029653
• License:
• Abstract: Large language models (LLMs) have become the preferred solution for many natural language processing tasks. In low-resource environments such as specialized domains, their few-shot capabilities are expected to deliver high performance. Named Entity Recognition (NER) is a critical task in information extraction that is not covered in recent LLM benchmarks. There is a need for better understanding the performance of LLMs for NER in a variety of settings including languages other than English. This study aims to evaluate generative LLMs, employed through prompt engineering, for few-shot clinical NER. %from the perspective of F1 performance and environmental impact. We compare 13 auto-regressive models using prompting and 16 masked models using fine-tuning on 14 NER datasets covering English, French and Spanish. While prompt-based auto-regressive models achieve competitive F1 for general NER, they are outperformed within the clinical domain by lighter biLSTM-CRF taggers based on masked models. Additionally, masked models exhibit lower environmental impact compared to auto-regressive models. Findings are consistent across the three languages studied, which suggests that LLM prompting is not yet suited for NER production in the clinical domain.

Related papers

• Enhancing Code Generation for Low-Resource Languages: No Silver Bullet [55.39571645315926]
  Large Language Models (LLMs) rely on large and diverse datasets to learn syntax, semantics, and usage patterns of programming languages. For low-resource languages, the limited availability of such data hampers the models' ability to generalize effectively. We present an empirical study investigating the effectiveness of several approaches for boosting LLMs' performance on low-resource languages.
  arXiv  Detail & Related papers  (2025-01-31T12:23:28Z)
• LLMic: Romanian Foundation Language Model [76.09455151754062]
  We present LLMic, a foundation language model designed specifically for the Romanian Language. We show that fine-tuning LLMic for language translation after the initial pretraining phase outperforms existing solutions in English-to-Romanian translation tasks.
  arXiv  Detail & Related papers  (2025-01-13T22:14:45Z)
• How to Learn a New Language? An Efficient Solution for Self-Supervised Learning Models Unseen Languages Adaption in Low-Resource Scenario [72.02391485962127]
  Speech Self-Supervised Learning (SSL) models achieve impressive performance on Automatic Speech Recognition (ASR) In low-resource language ASR, they encounter the domain mismatch problem between pre-trained and low-resource languages. We extend a conventional efficient fine-tuning scheme based on the adapter to handle these issues.
  arXiv  Detail & Related papers  (2024-11-27T10:51:00Z)
• GEIC: Universal and Multilingual Named Entity Recognition with Large Language Models [7.714969840571947]
  We introduce the task of generation-based extraction and in-context classification (GEIC) We then propose CascadeNER, a universal and multilingual GEIC framework for few-shot and zero-shot NER. We also introduce AnythingNER, the first NER dataset specifically designed for Large Language Models (LLMs)
  arXiv  Detail & Related papers  (2024-09-17T09:32:12Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• ProxyLM: Predicting Language Model Performance on Multilingual Tasks via Proxy Models [9.710960283117771]
  ProxyLM is a task- and language-agnostic framework designed to predict the performance of LMs using proxy models. ProxyLM significantly reduces computational overhead in task evaluations, achieving up to a 37.08x speedup over traditional methods. Our results demonstrate that ProxyLM not only adapts well to previously unseen languages in pre-trained LMs, but also generalizes effectively across different datasets.
  arXiv  Detail & Related papers  (2024-06-13T17:15:33Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• How far is Language Model from 100% Few-shot Named Entity Recognition in Medical Domain [14.635536657783613]
  This paper aims to compare the performance of LMs in medical few-shot NER and answer How far is LMs from 100% Few-shot NER in Medical Domain. Our findings clearly indicate that LLMs outperform SLMs in few-shot medical NER tasks, given the presence of suitable examples and appropriate logical frameworks. We introduce a simple and effective method called textscRT (Retrieving and Thinking), which serves as retrievers, finding relevant examples, and as thinkers, employing a step-by-step reasoning process.
  arXiv  Detail & Related papers  (2023-07-01T01:18:09Z)
• Extrapolating Multilingual Understanding Models as Multilingual Generators [82.1355802012414]
  This paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. We propose a textbfSemantic-textbfGuided textbfAlignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters.
  arXiv  Detail & Related papers  (2023-05-22T15:33:21Z)
• MEGA: Multilingual Evaluation of Generative AI [23.109803506475174]
  Generative AI models have shown impressive performance on many Natural Language Processing tasks. Most studies on generative LLMs have been restricted to English. It is unclear how capable these models are at understanding and generating text in other languages.
  arXiv  Detail & Related papers  (2023-03-22T13:03:10Z)
• MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning [12.640283469603357]
  Transformer language models (TLMs) are critical for most NLP tasks, but they are difficult to create for low-resource languages because of how much pretraining data they require. In this work, we investigate two techniques for training monolingual TLMs in a low-resource setting: greatly reducing TLM size, and complementing the masked language modeling objective with two linguistically rich supervised tasks. Results from 7 diverse languages indicate that our model, MicroBERT, is able to produce marked improvements in downstream task evaluations relative to a typical monolingual TLM pretraining approach.
  arXiv  Detail & Related papers  (2022-12-23T18:18:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.