Investigating Monolingual and Multilingual BERTModels for Vietnamese Aspect Category Detection

• URL: http://arxiv.org/abs/2103.09519v1
• Date: Wed, 17 Mar 2021 09:04:03 GMT
• Title: Investigating Monolingual and Multilingual BERTModels for Vietnamese Aspect Category Detection
• Authors: Dang Van Thin, Lac Si Le, Vu Xuan Hoang, Ngan Luu-Thuy Nguyen
• Abstract summary: This paper investigates the performance of various monolingual pre-trained language models compared with multilingual models on the Vietnamese aspect category detection problem. The experimental results demonstrated the effectiveness of the monolingual PhoBERT model than others on two datasets. To the best of our knowledge, our research study is the first attempt at performing various available pre-trained language models on aspect category detection task.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Aspect category detection (ACD) is one of the challenging tasks in the Aspect-based sentiment Analysis problem. The purpose of this task is to identify the aspect categories mentioned in user-generated reviews from a set of pre-defined categories. In this paper, we investigate the performance of various monolingual pre-trained language models compared with multilingual models on the Vietnamese aspect category detection problem. We conduct the experiments on two benchmark datasets for the restaurant and hotel domain. The experimental results demonstrated the effectiveness of the monolingual PhoBERT model than others on two datasets. We also evaluate the performance of the multilingual model based on the combination of whole SemEval-2016 datasets in other languages with the Vietnamese dataset. To the best of our knowledge, our research study is the first attempt at performing various available pre-trained language models on aspect category detection task and utilize the datasets from other languages based on multilingual models.

Related papers

• GradSim: Gradient-Based Language Grouping for Effective Multilingual Training [13.730907708289331]
  We propose GradSim, a language grouping method based on gradient similarity. Our experiments on three diverse multilingual benchmark datasets show that it leads to the largest performance gains. Besides linguistic features, the topics of the datasets play an important role for language grouping.
  arXiv  Detail & Related papers  (2023-10-23T18:13:37Z)
• Multilingual Few-Shot Learning via Language Model Retrieval [18.465566186549072]
  Transformer-based language models have achieved remarkable success in few-shot in-context learning. We conduct a study of retrieving semantically similar few-shot samples and using them as the context. We evaluate the proposed method on five natural language understanding datasets related to intent detection, question classification, sentiment analysis, and topic classification.
  arXiv  Detail & Related papers  (2023-06-19T14:27:21Z)
• Learning Cross-lingual Visual Speech Representations [108.68531445641769]
  Cross-lingual self-supervised visual representation learning has been a growing research topic in the last few years. We use the recently-proposed Raw Audio-Visual Speechs (RAVEn) framework to pre-train an audio-visual model with unlabelled data. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance.
  arXiv  Detail & Related papers  (2023-03-14T17:05:08Z)
• Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models [73.11488464916668]
  This study investigates the dynamics of the multilingual pretraining process. We probe checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones.
  arXiv  Detail & Related papers  (2022-05-24T03:35:00Z)
• A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification [16.684856745734944]
  We present a multilingual bag-of-entities model that boosts the performance of zero-shot cross-lingual text classification. It leverages the multilingual nature of Wikidata: entities in multiple languages representing the same concept are defined with a unique identifier. A model trained on entity features in a resource-rich language can thus be directly applied to other languages.
  arXiv  Detail & Related papers  (2021-10-15T01:10:50Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models [96.32118305166412]
  We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We find that languages which are adequately represented in the multilingual model's vocabulary exhibit negligible performance decreases over their monolingual counterparts.
  arXiv  Detail & Related papers  (2020-12-31T14:11:00Z)
• Indic-Transformers: An Analysis of Transformer Language Models for Indian Languages [0.8155575318208631]
  Language models based on the Transformer architecture have achieved state-of-the-art performance on a wide range of NLP tasks. However, this performance is usually tested and reported on high-resource languages, like English, French, Spanish, and German. Indian languages, on the other hand, are underrepresented in such benchmarks.
  arXiv  Detail & Related papers  (2020-11-04T14:43:43Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Mono vs Multilingual Transformer-based Models: a Comparison across Several Language Tasks [1.2691047660244335]
  BERT (Bidirectional Representations from Transformers) and ALBERT (A Lite BERT) are methods for pre-training language models. We make available our trained BERT and Albert model for Portuguese.
  arXiv  Detail & Related papers  (2020-07-19T19:13:20Z)
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization [128.37244072182506]
  Cross-lingual TRansfer Evaluation of Multilinguals XTREME is a benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models.
  arXiv  Detail & Related papers  (2020-03-24T19:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.