A Combination of BERT and Transformer for Vietnamese Spelling Correction

• URL: http://arxiv.org/abs/2405.02573v1
• Date: Sat, 4 May 2024 05:24:19 GMT
• Title: A Combination of BERT and Transformer for Vietnamese Spelling Correction
• Authors: Hieu Ngo Trung, Duong Tran Ham, Tin Huynh, Kiem Hoang,
• Abstract summary: There is no implementation in Vietnamese yet. Our model outperforms other approaches as well as the Google Docs Spell Checking tool, achieves an 86.24 BLEU score on this task.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, many studies have shown the efficiency of using Bidirectional Encoder Representations from Transformers (BERT) in various Natural Language Processing (NLP) tasks. Specifically, English spelling correction task that uses Encoder-Decoder architecture and takes advantage of BERT has achieved state-of-the-art result. However, to our knowledge, there is no implementation in Vietnamese yet. Therefore, in this study, a combination of Transformer architecture (state-of-the-art for Encoder-Decoder model) and BERT was proposed to deal with Vietnamese spelling correction. The experiment results have shown that our model outperforms other approaches as well as the Google Docs Spell Checking tool, achieves an 86.24 BLEU score on this task.

Related papers

• Dual-Alignment Pre-training for Cross-lingual Sentence Embedding [79.98111074307657]
  We propose a dual-alignment pre-training (DAP) framework for cross-lingual sentence embedding. We introduce a novel representation translation learning (RTL) task, where the model learns to use one-side contextualized token representation to reconstruct its translation counterpart. Our approach can significantly improve sentence embedding.
  arXiv  Detail & Related papers  (2023-05-16T03:53:30Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• XDBERT: Distilling Visual Information to BERT from Cross-Modal Systems to Improve Language Understanding [73.24847320536813]
  This study explores distilling visual information from pretrained multimodal transformers to pretrained language encoders. Our framework is inspired by cross-modal encoders' success in visual-language tasks while we alter the learning objective to cater to the language-heavy characteristics of NLU.
  arXiv  Detail & Related papers  (2022-04-15T03:44:00Z)
• Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding [57.08875260900373]
  We propose Shallow Aggressive Decoding (SAD) to improve the online inference efficiency of the Transformer for instantaneous Grammatical Error Correction (GEC) SAD aggressively decodes as many tokens as possible in parallel instead of always decoding only one token in each step to improve computational parallelism. Experiments in both English and Chinese GEC benchmarks show that aggressive decoding could yield the same predictions but with a significant speedup for online inference.
  arXiv  Detail & Related papers  (2021-06-09T10:30:59Z)
• Hierarchical Transformer Encoders for Vietnamese Spelling Correction [1.0779600811805266]
  We propose a Hierarchical Transformer model for Vietnamese spelling correction problem. The model consists of multiple Transformer encoders and utilizes both character-level and word-level to detect errors and make corrections.
  arXiv  Detail & Related papers  (2021-05-28T04:09:15Z)
• Explaining and Improving BERT Performance on Lexical Semantic Change Detection [22.934650688233734]
  Recent success of type-based models in SemEval-2020 Task 1 has raised the question why the success of token-based models does not translate to our field. We investigate the influence of a range of variables on clusterings of BERT vectors and show that its low performance is largely due to orthographic information on the target word.
  arXiv  Detail & Related papers  (2021-03-12T13:29:30Z)
• Fine-Tuning BERT for Sentiment Analysis of Vietnamese Reviews [0.0]
  Experimental results on two datasets show thatmodels using BERT slightly outperform other models usingGloVe and FastText. Our proposed BERT fine-tuning method produces amodel with better performance than the original BERT fine-tuning method.
  arXiv  Detail & Related papers  (2020-11-20T14:45:46Z)
• Adapting Pretrained Transformer to Lattices for Spoken Language Understanding [39.50831917042577]
  It is shown that encoding lattices as opposed to 1-best results generated by automatic speech recognizer (ASR) boosts the performance of spoken language understanding (SLU) This paper aims at adapting pretrained transformers to lattice inputs in order to perform understanding tasks specifically for spoken language.
  arXiv  Detail & Related papers  (2020-11-02T07:14:34Z)
• Incorporating BERT into Parallel Sequence Decoding with Adapters [82.65608966202396]
  We propose to take two different BERT models as the encoder and decoder respectively, and fine-tune them by introducing simple and lightweight adapter modules. We obtain a flexible and efficient model which is able to jointly leverage the information contained in the source-side and target-side BERT models. Our framework is based on a parallel sequence decoding algorithm named Mask-Predict considering the bi-directional and conditional independent nature of BERT.
  arXiv  Detail & Related papers  (2020-10-13T03:25:15Z)
• Information Extraction from Swedish Medical Prescriptions with Sig-Transformer Encoder [3.7921111379825088]
  We present a novel extension to the Transformer architecture, by incorporating signature transform with the self-attention model. Experiments on a new Swedish prescription data show the proposed architecture to be superior in two of the three information extraction tasks.
  arXiv  Detail & Related papers  (2020-10-10T04:22:07Z)
• Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation [59.38247587308604]
  We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation. We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T dataset. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models.
  arXiv  Detail & Related papers  (2020-03-30T21:35:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.