BERT for Sentiment Analysis: Pre-trained and Fine-Tuned Alternatives

• URL: http://arxiv.org/abs/2201.03382v1
• Date: Mon, 10 Jan 2022 15:05:05 GMT
• Title: BERT for Sentiment Analysis: Pre-trained and Fine-Tuned Alternatives
• Authors: Frederico Souza, Jo\~ao Filho
• Abstract summary: BERT has revolutionized the NLP field by enabling transfer learning with large language models. This article studies how to better cope with the different embeddings provided by the BERT output layer and the usage of language-specific instead of multilingual models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: BERT has revolutionized the NLP field by enabling transfer learning with large language models that can capture complex textual patterns, reaching the state-of-the-art for an expressive number of NLP applications. For text classification tasks, BERT has already been extensively explored. However, aspects like how to better cope with the different embeddings provided by the BERT output layer and the usage of language-specific instead of multilingual models are not well studied in the literature, especially for the Brazilian Portuguese language. The purpose of this article is to conduct an extensive experimental study regarding different strategies for aggregating the features produced in the BERT output layer, with a focus on the sentiment analysis task. The experiments include BERT models trained with Brazilian Portuguese corpora and the multilingual version, contemplating multiple aggregation strategies and open-source datasets with predefined training, validation, and test partitions to facilitate the reproducibility of the results. BERT achieved the highest ROC-AUC values for the majority of cases as compared to TF-IDF. Nonetheless, TF-IDF represents a good trade-off between the predictive performance and computational cost.

Related papers

• Comparison of Pre-trained Language Models for Turkish Address Parsing [0.0]
  We focus on Turkish maps data and thoroughly evaluate both multilingual and Turkish based BERT, DistilBERT, ELECTRA and RoBERTa. We also propose a MultiLayer Perceptron (MLP) for fine-tuning BERT in addition to the standard approach of one-layer fine-tuning.
  arXiv  Detail & Related papers  (2023-06-24T12:09:43Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding [73.24847320536813]
  This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders' success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU.
  arXiv  Detail & Related papers  (2022-04-15T03:44:00Z)
• Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation [50.54059385277964]
  We present a novel data augmentation paradigm termed Continuous Semantic Augmentation (CsaNMT) CsaNMT augments each training instance with an adjacency region that could cover adequate variants of literal expression under the same meaning.
  arXiv  Detail & Related papers  (2022-04-14T08:16:28Z)
• Fine-tuning Pretrained Multilingual BERT Model for Indonesian Aspect-based Sentiment Analysis [0.0]
  Previous research on Aspect-based Sentiment Analysis (ABSA) for Indonesian reviews in hotel domain has been conducted using CNN and XGBoost. In this paper, we intend to incorporate one of the foremost language representation model, BERT, to perform ABSA in Indonesian reviews dataset.
  arXiv  Detail & Related papers  (2021-03-05T15:05:51Z)
• Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization [56.87201892585477]
  Token embeddings in multilingual BERT (m-BERT) contain both language and semantic information. We control the output languages of multilingual BERT by manipulating the token embeddings.
  arXiv  Detail & Related papers  (2020-10-20T05:41:35Z)
• It's not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT [54.84185432755821]
  multilingual BERT (mBERT) learns rich cross-lingual representations, that allow for transfer across languages. We study the word-level translation information embedded in mBERT and present two simple methods that expose remarkable translation capabilities with no fine-tuning.
  arXiv  Detail & Related papers  (2020-10-16T09:49:32Z)
• Exploring Cross-sentence Contexts for Named Entity Recognition with BERT [1.4998865865537996]
  We present a study exploring the use of cross-sentence information for NER using BERT models in five languages. We find that adding context in the form of additional sentences to BERT input increases NER performance on all of the tested languages and models. We propose a straightforward method, Contextual Majority Voting (CMV), to combine different predictions for sentences and demonstrate this to further increase NER performance with BERT.
  arXiv  Detail & Related papers  (2020-06-02T12:34:52Z)
• Table Search Using a Deep Contextualized Language Model [20.041167804194707]
  In this paper, we use the deep contextualized language model BERT for the task of ad hoc table retrieval. We propose an approach that incorporates features from prior literature on table retrieval and jointly trains them with BERT.
  arXiv  Detail & Related papers  (2020-05-19T04:18:04Z)
• Cross-lingual Information Retrieval with BERT [8.052497255948046]
  We explore the use of the popular bidirectional language model, BERT, to model and learn the relevance between English queries and foreign-language documents. A deep relevance matching model based on BERT is introduced and trained by finetuning a pretrained multilingual BERT model with weak supervision. Experimental results of the retrieval of Lithuanian documents against short English queries show that our model is effective and outperforms the competitive baseline approaches.
  arXiv  Detail & Related papers  (2020-04-24T23:32:13Z)
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT [60.9051207862378]
  multilingual BERT works remarkably well on cross-lingual transfer tasks. Datasize and context window size are crucial factors to the transferability. There is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
  arXiv  Detail & Related papers  (2020-04-20T11:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.