Evaluating Named Entity Recognition: A comparative analysis of mono- and multilingual transformer models on a novel Brazilian corporate earnings call transcripts dataset

• URL: http://arxiv.org/abs/2403.12212v2
• Date: Fri, 30 Aug 2024 17:02:11 GMT
• Title: Evaluating Named Entity Recognition: A comparative analysis of mono- and multilingual transformer models on a novel Brazilian corporate earnings call transcripts dataset
• Authors: Ramon Abilio, Guilherme Palermo Coelho, Ana Estela Antunes da Silva,
• Abstract summary: We identify two models pre-trained in Brazilian Portuguese (BERTimbau and PTT5) and two multilingual models (mBERT and mT5) Our study aimed to evaluate their performance on a financial Named Entity Recognition (NER) task and determine the computational requirements for fine-tuning and inference.
• Score: 3.809702129519642
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Since 2018, when the Transformer architecture was introduced, Natural Language Processing has gained significant momentum with pre-trained Transformer-based models that can be fine-tuned for various tasks. Most models are pre-trained on large English corpora, making them less applicable to other languages, such as Brazilian Portuguese. In our research, we identified two models pre-trained in Brazilian Portuguese (BERTimbau and PTT5) and two multilingual models (mBERT and mT5). BERTimbau and mBERT use only the Encoder module, while PTT5 and mT5 use both the Encoder and Decoder. Our study aimed to evaluate their performance on a financial Named Entity Recognition (NER) task and determine the computational requirements for fine-tuning and inference. To this end, we developed the Brazilian Financial NER (BraFiNER) dataset, comprising sentences from Brazilian banks' earnings calls transcripts annotated using a weakly supervised approach. Additionally, we introduced a novel approach that reframes the token classification task as a text generation problem. After fine-tuning the models, we evaluated them using performance and error metrics. Our findings reveal that BERT-based models consistently outperform T5-based models. While the multilingual models exhibit comparable macro F1-scores, BERTimbau demonstrates superior performance over PTT5. In terms of error metrics, BERTimbau outperforms the other models. We also observed that PTT5 and mT5 generated sentences with changes in monetary and percentage values, highlighting the importance of accuracy and consistency in the financial domain. Our findings provide insights into the differing performance of BERT- and T5-based models for the NER task.

Related papers

• BERT vs GPT for financial engineering [0.0]
  The paper benchmarks several Transformer models to show how these models can judge sentiment from a news event. We find that fine-tuned BERT models outperform fine-tuned or vanilla GPT models on this task.
  arXiv  Detail & Related papers  (2024-04-24T11:30:04Z)
• A Text-to-Text Model for Multilingual Offensive Language Identification [19.23565690468299]
  This study presents the first pre-trained model with encoder-decoder architecture for offensive language identification with text-to-text transformers (T5) Our pre-trained T5 model outperforms other transformer-based models fine-tuned for offensive language detection, such as fBERT and HateBERT, in multiple English benchmarks. Following a similar approach, we also train the first multilingual pre-trained model for offensive language identification using mT5.
  arXiv  Detail & Related papers  (2023-12-06T09:37:27Z)
• On Robustness of Finetuned Transformer-based NLP Models [11.063628128069736]
  We characterize changes between pretrained and finetuned language model representations across layers using two metrics: CKA and STIR. GPT-2 representations are more robust than BERT and T5 across multiple types of input perturbations. This study provides valuable insights into perturbation-specific weaknesses of popular Transformer-based models.
  arXiv  Detail & Related papers  (2023-05-23T18:25:18Z)
• Transformer-based approaches to Sentiment Detection [55.41644538483948]
  We examined the performance of four different types of state-of-the-art transformer models for text classification. The RoBERTa transformer model performs best on the test dataset with a score of 82.6% and is highly recommended for quality predictions.
  arXiv  Detail & Related papers  (2023-03-13T17:12:03Z)
• Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation [9.736284584478032]
  We show the effectiveness of character-level modeling in translation, particularly in cases where fine-tuning data is limited. While evaluating the importance of source texts in driving model predictions, we highlight word-level patterns within ByT5. We conclude by assessing the efficiency tradeoff of byte models, suggesting their usage in non-time-critical scenarios to boost translation quality.
  arXiv  Detail & Related papers  (2023-02-28T00:50:19Z)
• idT5: Indonesian Version of Multilingual T5 Transformer [0.0]
  Indonesian is spoken by almost 200 million people and is the 10th most spoken language in the world. In this study, the mT5 model was adapted for only one language, Indonesian, resulting in a pre-trained T5 model that was specific only for Indonesian with a smaller size. Fine-tuned model based on our model achieved 77.18% accuracy on SA, 8% higher than the mT5-based model, and obtained nearly the same score as the mT5-based model on QG and QA.
  arXiv  Detail & Related papers  (2023-02-02T03:56:16Z)
• Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysis [84.12658971655253]
  We propose Adapted Multimodal BERT, a BERT-based architecture for multimodal tasks. adapter adjusts the pretrained language model for the task at hand, while the fusion layers perform task-specific, layer-wise fusion of audio-visual information with textual BERT representations. In our ablations we see that this approach leads to efficient models, that can outperform their fine-tuned counterparts and are robust to input noise.
  arXiv  Detail & Related papers  (2022-12-01T17:31:42Z)
• Evaluation of Transfer Learning for Polish with a Text-to-Text Model [54.81823151748415]
  We introduce a new benchmark for assessing the quality of text-to-text models for Polish. The benchmark consists of diverse tasks and datasets: KLEJ benchmark adapted for text-to-text, en-pl translation, summarization, and question answering. We present plT5 - a general-purpose text-to-text model for Polish that can be fine-tuned on various Natural Language Processing (NLP) tasks with a single training objective.
  arXiv  Detail & Related papers  (2022-05-18T09:17:14Z)
• Few-shot learning through contextual data augmentation [74.20290390065475]
  Machine translation models need to adapt to new data to maintain their performance over time. We show that adaptation on the scale of one to five examples is possible. Our model reports better accuracy scores than a reference system trained with on average 313 parallel examples.
  arXiv  Detail & Related papers  (2021-03-31T09:05:43Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• mT5: A massively multilingual pre-trained text-to-text transformer [60.0210636815514]
  "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on English-language NLP tasks. We introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages.
  arXiv  Detail & Related papers  (2020-10-22T17:58:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.