Spelling Correction with Denoising Transformer

• URL: http://arxiv.org/abs/2105.05977v1
• Date: Wed, 12 May 2021 21:35:18 GMT
• Title: Spelling Correction with Denoising Transformer
• Authors: Alex Kuznetsov, Hector Urdiales
• Abstract summary: We present a novel method of performing spelling correction on short input strings, such as search queries or individual words. At its core lies a procedure for generating artificial typos which closely follow the error patterns manifested by humans. This procedure is used to train the production spelling correction model based on a transformer architecture.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a novel method of performing spelling correction on short input strings, such as search queries or individual words. At its core lies a procedure for generating artificial typos which closely follow the error patterns manifested by humans. This procedure is used to train the production spelling correction model based on a transformer architecture. This model is currently served in the HubSpot product search. We show that our approach to typo generation is superior to the widespread practice of adding noise, which ignores human patterns. We also demonstrate how our approach may be extended to resource-scarce settings and train spelling correction models for Arabic, Greek, Russian, and Setswana languages, without using any labeled data.

Related papers

• A Methodology for Generative Spelling Correction via Natural Spelling Errors Emulation across Multiple Domains and Languages [39.75847219395984]
  We present a methodology for generative spelling correction (SC), which was tested on English and Russian languages. We study the ways those errors can be emulated in correct sentences to effectively enrich generative models' pre-train procedure. As a practical outcome of our work, we introduce SAGE(Spell checking via Augmentation and Generative distribution Emulation)
  arXiv  Detail & Related papers  (2023-08-18T10:07:28Z)
• Chinese Spelling Correction as Rephrasing Language Model [63.65217759957206]
  We study Chinese Spelling Correction (CSC), which aims to detect and correct the potential spelling errors in a given sentence. Current state-of-the-art methods regard CSC as a sequence tagging task and fine-tune BERT-based models on sentence pairs. We propose Rephrasing Language Model (ReLM), where the model is trained to rephrase the entire sentence by infilling additional slots, instead of character-to-character tagging.
  arXiv  Detail & Related papers  (2023-08-17T06:04:28Z)
• An Error-Guided Correction Model for Chinese Spelling Error Correction [13.56600372085612]
  We propose an error-guided correction model (EGCM) to improve Chinese spelling correction. Our model achieves superior performance against state-of-the-art approaches by a remarkable margin.
  arXiv  Detail & Related papers  (2023-01-16T09:27:45Z)
• Sentence Bottleneck Autoencoders from Transformer Language Models [53.350633961266375]
  We build a sentence-level autoencoder from a pretrained, frozen transformer language model. We adapt the masked language modeling objective as a generative, denoising one, while only training a sentence bottleneck and a single-layer modified transformer decoder. We demonstrate that the sentence representations discovered by our model achieve better quality than previous methods that extract representations from pretrained transformers on text similarity tasks, style transfer, and single-sentence classification tasks in the GLUE benchmark, while using fewer parameters than large pretrained models.
  arXiv  Detail & Related papers  (2021-08-31T19:39:55Z)
• Hierarchical Transformer Encoders for Vietnamese Spelling Correction [1.0779600811805266]
  We propose a Hierarchical Transformer model for Vietnamese spelling correction problem. The model consists of multiple Transformer encoders and utilizes both character-level and word-level to detect errors and make corrections.
  arXiv  Detail & Related papers  (2021-05-28T04:09:15Z)
• Grammatical Error Generation Based on Translated Fragments [0.0]
  We perform neural machine translation of sentence fragments in order to create large amounts of training data for English grammatical error correction. Our method aims at simulating mistakes made by second language learners, and produces a wider range of non-native style language.
  arXiv  Detail & Related papers  (2021-04-20T12:43:40Z)
• Improving Translation Robustness with Visual Cues and Error Correction [58.97421756225425]
  We introduce the idea of visual context to improve translation robustness against noisy texts. We also propose a novel error correction training regime by treating error correction as an auxiliary task.
  arXiv  Detail & Related papers  (2021-03-12T15:31:34Z)
• NeuSpell: A Neural Spelling Correction Toolkit [88.79419580807519]
  NeuSpell is an open-source toolkit for spelling correction in English. It comprises ten different models, and benchmarks them on misspellings from multiple sources. We train neural models using spelling errors in context, synthetically constructed by reverse engineering isolated misspellings.
  arXiv  Detail & Related papers  (2020-10-21T15:53:29Z)
• Word Shape Matters: Robust Machine Translation with Visual Embedding [78.96234298075389]
  We introduce a new encoding of the input symbols for character-level NLP models. It encodes the shape of each character through the images depicting the letters when printed. We name this new strategy visual embedding and it is expected to improve the robustness of NLP models.
  arXiv  Detail & Related papers  (2020-10-20T04:08:03Z)
• On the Robustness of Language Encoders against Grammatical Errors [66.05648604987479]
  We collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. Results confirm that the performance of all tested models is affected but the degree of impact varies.
  arXiv  Detail & Related papers  (2020-05-12T11:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.