Disentangling Homophemes in Lip Reading using Perplexity Analysis

• URL: http://arxiv.org/abs/2012.07528v1
• Date: Sat, 28 Nov 2020 12:12:17 GMT
• Title: Disentangling Homophemes in Lip Reading using Perplexity Analysis
• Authors: Souheil Fenghour, Daqing Chen, Kun Guo, Perry Xiao
• Abstract summary: This paper proposes a new application for the Generative Pre-Training transformer. It serves as a language model to convert visual speech in the form of visemes, to language in the form of words and sentences. The network uses the search for optimal perplexity to perform the viseme-to-word mapping.
• Score: 10.262299768603894
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The performance of automated lip reading using visemes as a classification schema has achieved less success compared with the use of ASCII characters and words largely due to the problem of different words sharing identical visemes. The Generative Pre-Training transformer is an effective autoregressive language model used for many tasks in Natural Language Processing, including sentence prediction and text classification. This paper proposes a new application for this model and applies it in the context of lip reading, where it serves as a language model to convert visual speech in the form of visemes, to language in the form of words and sentences. The network uses the search for optimal perplexity to perform the viseme-to-word mapping and is thus a solution to the one-to-many mapping problem that exists whereby various words that sound different when spoken look identical. This paper proposes a method to tackle the one-to-many mapping problem when performing automated lip reading using solely visual cues in two separate scenarios: the first scenario is where the word boundary, that is, the beginning and the ending of a word, is unknown; and the second scenario is where the boundary is known. Sentences from the benchmark BBC dataset "Lip Reading Sentences in the Wild"(LRS2), are classified with a character error rate of 10.7% and a word error rate of 18.0%. The main contribution of this paper is to propose a method of predicting words through the use of perplexity analysis when only visual cues are present, using an autoregressive language model.

Related papers

• CAST: Corpus-Aware Self-similarity Enhanced Topic modelling [16.562349140796115]
  We introduce CAST: Corpus-Aware Self-similarity Enhanced Topic modelling, a novel topic modelling method. We find self-similarity to be an effective metric to prevent functional words from acting as candidate topic words. Our approach significantly enhances the coherence and diversity of generated topics, as well as the topic model's ability to handle noisy data.
  arXiv  Detail & Related papers  (2024-10-19T15:27:11Z)
• Fine-tuning CLIP Text Encoders with Two-step Paraphrasing [83.3736789315201]
  We introduce a straightforward fine-tuning approach to enhance the representations of CLIP models for paraphrases. Our model, which we call ParaCLIP, exhibits significant improvements over baseline CLIP models across various tasks.
  arXiv  Detail & Related papers  (2024-02-23T06:11:50Z)
• Is it an i or an l: Test-time Adaptation of Text Line Recognition Models [9.149602257966917]
  We introduce the problem of adapting text line recognition models during test time. We propose an iterative self-training approach that uses feedback from the language model to update the optical model. Experimental results show that the proposed adaptation method offers an absolute improvement of up to 8% in character error rate.
  arXiv  Detail & Related papers  (2023-08-29T05:44:00Z)
• Integrating Bidirectional Long Short-Term Memory with Subword Embedding for Authorship Attribution [2.3429306644730854]
  Manifold word-based stylistic markers have been successfully used in deep learning methods to deal with the intrinsic problem of authorship attribution. The proposed method was experimentally evaluated against numerous state-of-the-art methods across the public corporal of CCAT50, IMDb62, Blog50, and Twitter50.
  arXiv  Detail & Related papers  (2023-06-26T11:35:47Z)
• Improving Contextual Recognition of Rare Words with an Alternate Spelling Prediction Model [0.0]
  We release contextual biasing lists to accompany the Earnings21 dataset. We show results for shallow fusion contextual biasing applied to two different decoding algorithms. We propose an alternate spelling prediction model that improves recall of rare words by 34.7% relative.
  arXiv  Detail & Related papers  (2022-09-02T19:30:16Z)
• Sub-word Level Lip Reading With Visual Attention [88.89348882036512]
  We focus on the unique challenges encountered in lip reading and propose tailored solutions. We obtain state-of-the-art results on the challenging LRS2 and LRS3 benchmarks when training on public datasets. Our best model achieves 22.6% word error rate on the LRS2 dataset, a performance unprecedented for lip reading models.
  arXiv  Detail & Related papers  (2021-10-14T17:59:57Z)
• Fake it Till You Make it: Self-Supervised Semantic Shifts for Monolingual Word Embedding Tasks [58.87961226278285]
  We propose a self-supervised approach to model lexical semantic change. We show that our method can be used for the detection of semantic change with any alignment method. We illustrate the utility of our techniques using experimental results on three different datasets.
  arXiv  Detail & Related papers  (2021-01-30T18:59:43Z)
• ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Models [0.0]
  Deep learning approaches based on CNN, LSTM, and Transformers have been the de facto approach for text classification. We show that these systems are over-reliant on the important words present in the text that are useful for classification.
  arXiv  Detail & Related papers  (2021-01-30T15:18:35Z)
• Match-Ignition: Plugging PageRank into Transformer for Long-form Text Matching [66.71886789848472]
  We propose a novel hierarchical noise filtering model, namely Match-Ignition, to tackle the effectiveness and efficiency problem. The basic idea is to plug the well-known PageRank algorithm into the Transformer, to identify and filter both sentence and word level noisy information. Noisy sentences are usually easy to detect because the sentence is the basic unit of a long-form text, so we directly use PageRank to filter such information.
  arXiv  Detail & Related papers  (2021-01-16T10:34:03Z)
• NLP-CIC @ DIACR-Ita: POS and Neighbor Based Distributional Models for Lexical Semantic Change in Diachronic Italian Corpora [62.997667081978825]
  We present our systems and findings on unsupervised lexical semantic change for the Italian language. The task is to determine whether a target word has evolved its meaning with time, only relying on raw-text from two time-specific datasets. We propose two models representing the target words across the periods to predict the changing words using threshold and voting schemes.
  arXiv  Detail & Related papers  (2020-11-07T11:27:18Z)
• Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification [52.69730591919885]
  We present a semi-supervised adversarial training process that minimizes the maximal loss for label-preserving input perturbations. We observe significant gains in effectiveness on document and intent classification for a diverse set of languages.
  arXiv  Detail & Related papers  (2020-07-29T19:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.