Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency

• URL: http://arxiv.org/abs/2306.04530v1
• Date: Wed, 7 Jun 2023 15:39:02 GMT
• Title: Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency
• Authors: Shigeki Karita, Richard Sproat, Haruko Ishikawa
• Abstract summary: We create a lattice of plausible respellings of the reference transcription using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model. Our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%-3.1% absolute reduction in CER depending on the task.
• Score: 8.888638284299736
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Word error rate (WER) and character error rate (CER) are standard metrics in Speech Recognition (ASR), but one problem has always been alternative spellings: If one's system transcribes adviser whereas the ground truth has advisor, this will count as an error even though the two spellings really represent the same word. Japanese is notorious for ``lacking orthography'': most words can be spelled in multiple ways, presenting a problem for accurate ASR evaluation. In this paper we propose a new lenient evaluation metric as a more defensible CER measure for Japanese ASR. We create a lattice of plausible respellings of the reference transcription, using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model for reconstructing kanji from hiragana or katakana. In a manual evaluation, raters rated 95.4% of the proposed spelling variants as plausible. ASR results show that our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%-3.1% absolute reduction in CER depending on the task.

Related papers

• JaPOC: Japanese Post-OCR Correction Benchmark using Vouchers [0.0]
  We create benchmarks and assess the effectiveness of error correction methods for Japanese vouchers in OCR (Optical Character Recognition) systems. In the experiments, the proposed error correction algorithm significantly improved overall recognition accuracy.
  arXiv  Detail & Related papers  (2024-09-30T05:01:49Z)
• A Coin Has Two Sides: A Novel Detector-Corrector Framework for Chinese Spelling Correction [79.52464132360618]
  Chinese Spelling Correction (CSC) stands as a foundational Natural Language Processing (NLP) task. We introduce a novel approach based on error detector-corrector framework. Our detector is designed to yield two error detection results, each characterized by high precision and recall.
  arXiv  Detail & Related papers  (2024-09-06T09:26:45Z)
• Automatic Real-word Error Correction in Persian Text [0.0]
  This paper introduces a cutting-edge approach for precise and efficient real-word error correction in Persian text. We employ semantic analysis, feature selection, and advanced classifiers to enhance error detection and correction efficacy. Our method achieves an impressive F-measure of 96.6% in the detection phase and an accuracy of 99.1% in the correction phase.
  arXiv  Detail & Related papers  (2024-07-20T07:50:52Z)
• Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition [52.624909026294105]
  We propose a non-autoregressive speech error correction method. A Confidence Module measures the uncertainty of each word of the N-best ASR hypotheses. The proposed system reduces the error rate by 21% compared with the ASR model.
  arXiv  Detail & Related papers  (2024-06-29T17:56:28Z)
• Chinese Spelling Correction as Rephrasing Language Model [63.65217759957206]
  We study Chinese Spelling Correction (CSC), which aims to detect and correct the potential spelling errors in a given sentence. Current state-of-the-art methods regard CSC as a sequence tagging task and fine-tune BERT-based models on sentence pairs. We propose Rephrasing Language Model (ReLM), where the model is trained to rephrase the entire sentence by infilling additional slots, instead of character-to-character tagging.
  arXiv  Detail & Related papers  (2023-08-17T06:04:28Z)
• SpellMapper: A non-autoregressive neural spellchecker for ASR customization with candidate retrieval based on n-gram mappings [76.87664008338317]
  Contextual spelling correction models are an alternative to shallow fusion to improve automatic speech recognition. We propose a novel algorithm for candidate retrieval based on misspelled n-gram mappings. Experiments on Spoken Wikipedia show 21.4% word error rate improvement compared to a baseline ASR system.
  arXiv  Detail & Related papers  (2023-06-04T10:00:12Z)
• Unsupervised Language agnostic WER Standardization [4.768240090076601]
  We propose an automatic WER normalization system consisting of two modules: spelling normalization and segmentation normalization. Experiments with ASR on 35K utterances across four languages yielded an average WER reduction of 13.28%.
  arXiv  Detail & Related papers  (2023-03-09T05:50:54Z)
• End-to-End Page-Level Assessment of Handwritten Text Recognition [69.55992406968495]
  HTR systems increasingly face the end-to-end page-level transcription of a document. Standard metrics do not take into account the inconsistencies that might appear. We propose a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately.
  arXiv  Detail & Related papers  (2023-01-14T15:43:07Z)
• Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement [57.72846454929923]
  We create a benchmark dataset, emphHJQE, where the expert translators directly annotate poorly translated words. We propose two tag correcting strategies, namely tag refinement strategy and tree-based annotation strategy, to make the TER-based artificial QE corpus closer to emphHJQE. The results show our proposed dataset is more consistent with human judgement and also confirm the effectiveness of the proposed tag correcting strategies.
  arXiv  Detail & Related papers  (2022-09-13T02:37:12Z)
• Is Word Error Rate a good evaluation metric for Speech Recognition in Indic Languages? [0.0]
  We propose a new method for the calculation of error rates in Automatic Speech Recognition (ASR) This new metric is for languages that contain half characters and where the same character can be written in different forms. We implement our methodology in Hindi which is one of the main languages from Indic context.
  arXiv  Detail & Related papers  (2022-03-30T18:32:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.