TweetBERT: A Pretrained Language Representation Model for Twitter Text Analysis

• URL: http://arxiv.org/abs/2010.11091v1
• Date: Sat, 17 Oct 2020 00:45:02 GMT
• Title: TweetBERT: A Pretrained Language Representation Model for Twitter Text Analysis
• Authors: Mohiuddin Md Abdul Qudar, Vijay Mago
• Abstract summary: We introduce two TweetBERT models, which are domain specific language presentation models, pre-trained on millions of tweets. We show that the TweetBERT models significantly outperform the traditional BERT models in Twitter text mining tasks by more than 7% on each Twitter dataset.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Twitter is a well-known microblogging social site where users express their views and opinions in real-time. As a result, tweets tend to contain valuable information. With the advancements of deep learning in the domain of natural language processing, extracting meaningful information from tweets has become a growing interest among natural language researchers. Applying existing language representation models to extract information from Twitter does not often produce good results. Moreover, there is no existing language representation models for text analysis specific to the social media domain. Hence, in this article, we introduce two TweetBERT models, which are domain specific language presentation models, pre-trained on millions of tweets. We show that the TweetBERT models significantly outperform the traditional BERT models in Twitter text mining tasks by more than 7% on each Twitter dataset. We also provide an extensive analysis by evaluating seven BERT models on 31 different datasets. Our results validate our hypothesis that continuously training language models on twitter corpus help performance with Twitter.

Related papers

• RoBERTweet: A BERT Language Model for Romanian Tweets [0.15293427903448023]
  This article introduces RoBERTweet, the first Transformer architecture trained on Romanian tweets. The corpus used for pre-training the models represents a novelty for the Romanian NLP community. Experiments show that RoBERTweet models outperform the previous general-domain Romanian and multilingual language models on three NLP tasks with tweet inputs.
  arXiv  Detail & Related papers  (2023-06-11T06:11:56Z)
• TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter [31.698196219228024]
  We present TwHIN-BERT, a multilingual language model productionized at Twitter. Our model is trained on 7 billion tweets covering over 100 distinct languages. We evaluate our model on various multilingual social recommendation and semantic understanding tasks.
  arXiv  Detail & Related papers  (2022-09-15T19:01:21Z)
• ViralBERT: A User Focused BERT-Based Approach to Virality Prediction [11.992815669875924]
  We propose ViralBERT, which can be used to predict the virality of tweets using content- and user-based features. We employ a method of concatenating numerical features such as hashtags and follower numbers to tweet text, and utilise two BERT modules. We collect a dataset of 330k tweets to train ViralBERT and validate the efficacy of our model using baselines from current studies in this field.
  arXiv  Detail & Related papers  (2022-05-17T21:40:24Z)
• BERTuit: Understanding Spanish language in Twitter through a native transformer [70.77033762320572]
  We present bfBERTuit, the larger transformer proposed so far for Spanish language, pre-trained on a massive dataset of 230M Spanish tweets. Our motivation is to provide a powerful resource to better understand Spanish Twitter and to be used on applications focused on this social network.
  arXiv  Detail & Related papers  (2022-04-07T14:28:51Z)
• Exploiting BERT For Multimodal Target SentimentClassification Through Input Space Translation [75.82110684355979]
  We introduce a two-stream model that translates images in input space using an object-aware transformer. We then leverage the translation to construct an auxiliary sentence that provides multimodal information to a language model. We achieve state-of-the-art performance on two multimodal Twitter datasets.
  arXiv  Detail & Related papers  (2021-08-03T18:02:38Z)
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models [59.107260266206445]
  Short texts published on Twitter have earned significant attention as a rich source of information. Their inherent characteristics, such as the informal, and noisy linguistic style, remain challenging to many natural language processing (NLP) tasks. This study fulfils an assessment of existing language models in distinguishing the sentiment expressed in tweets by using a rich collection of 22 datasets.
  arXiv  Detail & Related papers  (2021-05-29T21:05:28Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Improving Sentiment Analysis over non-English Tweets using Multilingual Transformers and Automatic Translation for Data-Augmentation [77.69102711230248]
  We propose the use of a multilingual transformer model, that we pre-train over English tweets and apply data-augmentation using automatic translation to adapt the model to non-English languages. Our experiments in French, Spanish, German and Italian suggest that the proposed technique is an efficient way to improve the results of the transformers over small corpora of tweets in a non-English language.
  arXiv  Detail & Related papers  (2020-10-07T15:44:55Z)
• InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective [84.78604733927887]
  Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models.
  arXiv  Detail & Related papers  (2020-10-05T20:49:26Z)
• Sentiment Analysis on Social Media Content [0.0]
  The aim of this paper is to present a model that can perform sentiment analysis of real data collected from Twitter. Data in Twitter is highly unstructured which makes it difficult to analyze. Our proposed model is different from prior work in this field because it combined the use of supervised and unsupervised machine learning algorithms.
  arXiv  Detail & Related papers  (2020-07-04T17:03:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.