Clinical named entity recognition in the Portuguese language: a benchmark of modern BERT models and LLMs

• URL: http://arxiv.org/abs/2603.26510v1
• Date: Fri, 27 Mar 2026 15:22:07 GMT
• Title: Clinical named entity recognition in the Portuguese language: a benchmark of modern BERT models and LLMs
• Abstract summary: We evaluate BERT-based models and large language models (LLMs) for clinical NER in Portuguese.<n>We compared BioBERTpt, BERTimbau, ModernBERT, and mmBERT with LLMs such as GPT-5 and Gemini-2.5.<n>The mmBERT-base model achieved the best performance (micro F1 = 0.76), outperforming all other models.
• Score: 1.6114494927075578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical notes contain valuable unstructured information. Named entity recognition (NER) enables the automatic extraction of medical concepts; however, benchmarks for Portuguese remain scarce. In this study, we aimed to evaluate BERT-based models and large language models (LLMs) for clinical NER in Portuguese and to test strategies for addressing multilabel imbalance. We compared BioBERTpt, BERTimbau, ModernBERT, and mmBERT with LLMs such as GPT-5 and Gemini-2.5, using the public SemClinBr corpus and a private breast cancer dataset. Models were trained under identical conditions and evaluated using precision, recall, and F1-score. Iterative stratification, weighted loss, and oversampling were explored to mitigate class imbalance. The mmBERT-base model achieved the best performance (micro F1 = 0.76), outperforming all other models. Iterative stratification improved class balance and overall performance. Multilingual BERT models, particularly mmBERT, perform strongly for Portuguese clinical NER and can run locally with limited computational resources. Balanced data-splitting strategies further enhance performance.

Related papers

• Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders [42.78591555984395]
  Portuguese is one of the most widely spoken languages in the world.<n> embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear.<n>We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB.
  arXiv  Detail & Related papers  (2026-07-05T01:06:29Z)
• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports [49.5225801722164]
  Existing metrics obscure this requirement by reducing report quality to a medically ungrounded scalar.<n>We study this boundary using ReEvalMed benchmark as testbed and evaluate metric-level clinical significance.
  arXiv  Detail & Related papers  (2026-06-17T08:10:30Z)
• The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance [5.94491873775155]
  InferBERT framework integrates transformer models with Do-calculus.<n>Four models were evaluated-XGBoost (baseline), ALBERT, BioBERT, and Med-LLaMA.<n> domain-specific pre-training was decisive.<n>BioBERT achieved the highest accuracy on both datasets.
  arXiv  Detail & Related papers  (2026-06-15T09:01:49Z)
• Beyond the Basics: Leveraging Large Language Model for Fine-Grained Medical Entity Recognition [0.12189422792863447]
  We rigorously evaluated the open-source LLaMA3 model for fine-grained medical entity recognition.<n>We employed three learning paradigms: zero-shot, few-shot, and fine-tuning with Low-Rank Adaptation (LoRA)<n>Our results showed that fine-tuned LLaMA3 surpasses zero-shot and few-shot approaches by 63.11% and 35.63%, respectivel respectively.
  arXiv  Detail & Related papers  (2026-04-19T02:50:14Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• Information Extraction from Clinical Notes: Are We Ready to Switch to Large Language Models? [16.312594953592665]
  Large language models (LLMs) excel on generative tasks, but their performance on extractive tasks remains debated.<n>This study is among the first to develop and evaluate a comprehensive clinical IE system using open-source LLMs.
  arXiv  Detail & Related papers  (2024-11-15T07:54:19Z)
• WellDunn: On the Robustness and Explainability of Language Models and Large Language Models in Identifying Wellness Dimensions [46.60244609728416]
  Language Models (LMs) are being proposed for mental health applications where the heightened risk of adverse outcomes means predictive performance may not be a litmus test of a model's utility in clinical practice. We introduce an evaluation design that focuses on the robustness and explainability of LMs in identifying Wellness Dimensions (WDs) We reveal four surprising results about LMs/LLMs.
  arXiv  Detail & Related papers  (2024-06-17T19:50:40Z)
• A Benchmark Evaluation of Clinical Named Entity Recognition in French [4.430193084761607]
  We evaluate biomedical models CamemBERT-bio and DrBERT and compare them to standard French models CamemBERT, FlauBERT and FrALBERT. Results suggest that CamemBERT-bio outperforms DrBERT consistently while FlauBERT offers competitive performance and FrAlBERT achieves the lowest carbonprint.
  arXiv  Detail & Related papers  (2024-03-28T07:59:58Z)
• Exploring the Value of Pre-trained Language Models for Clinical Named Entity Recognition [6.917786124918387]
  We compare Transformer models that are trained from scratch to fine-tuned BERT-based LLMs. We examine the impact of an additional CRF layer on such models to encourage contextual learning.
  arXiv  Detail & Related papers  (2022-10-23T16:27:31Z)
• Learning structures of the French clinical language:development and validation of word embedding models using 21 million clinical reports from electronic health records [2.5709272341038027]
  Methods based on transfer learning using pre-trained language models have achieved state-of-the-art results in most NLP applications. We aimed to evaluate the impact of adapting a language model to French clinical reports on downstream medical NLP tasks.
  arXiv  Detail & Related papers  (2022-07-26T14:46:34Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing [55.52858954615655]
  We conduct a systematic study on fine-tuning stability in biomedical NLP. We show that finetuning performance may be sensitive to pretraining settings, especially in low-resource domains. We show that these techniques can substantially improve fine-tuning performance for lowresource biomedical NLP applications.
  arXiv  Detail & Related papers  (2021-12-15T04:20:35Z)
• Performance of Dual-Augmented Lagrangian Method and Common Spatial Patterns applied in classification of Motor-Imagery BCI [68.8204255655161]
  Motor-imagery based brain-computer interfaces (MI-BCI) have the potential to become ground-breaking technologies for neurorehabilitation. Due to the noisy nature of the used EEG signal, reliable BCI systems require specialized procedures for features optimization and extraction.
  arXiv  Detail & Related papers  (2020-10-13T20:50:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.