Lattice-Free MMI Adaptation Of Self-Supervised Pretrained Acoustic Models

• URL: http://arxiv.org/abs/2012.14252v2
• Date: Tue, 6 Apr 2021 14:41:54 GMT
• Title: Lattice-Free MMI Adaptation Of Self-Supervised Pretrained Acoustic Models
• Abstract summary: We propose lattice-free MMI (LFMMI) for supervised adaptation of self-supervised pretrained acoustic model. Our results show that fine-tuning with LFMMI, we consistently obtain relative WER improvements of 10% and 35.3%.
• Score: 6.338178373376447
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we propose lattice-free MMI (LFMMI) for supervised adaptation of self-supervised pretrained acoustic model. We pretrain a Transformer model on thousand hours of untranscribed Librispeech data followed by supervised adaptation with LFMMI on three different datasets. Our results show that fine-tuning with LFMMI, we consistently obtain relative WER improvements of 10% and 35.3% on the clean and other test sets of Librispeech (100h), 10.8% on Switchboard (300h), and 4.3% on Swahili (38h) and 4.4% on Tagalog (84h) compared to the baseline trained only with supervised data.

Related papers

• TEMPO: Scaling Test-time Training for Large Reasoning Models [87.61789183311856]
  Test-time training (TTT) adapts model parameters on unlabeled test instances during inference time.<n>We propose TEMPO, a TTT framework that interleaves policy refinement on unlabeled questions with periodic critic recalibration on a labeled dataset.
  arXiv  Detail & Related papers  (2026-04-21T10:01:04Z)
• Muon: Training and Trade-offs with Latent Attention and MoE [4.500362688166346]
  We present a comprehensive theoretical and empirical study of the Muon for training transformers only with a small to medium decoder (30M - 200M parameters)<n>We provide rigorous theoretical analysis including: (i)showing the convergence rate under standard assumptions, (ii) spectral regularization properties that prevent gradient explosion, (iii) connection to natural gradient descent on the Stiefel manifold, and (iv) equivalence to steepest gradient descent under the spectral norm.
  arXiv  Detail & Related papers  (2025-09-29T07:51:06Z)
• Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels [72.3670919950349]
  Large language models (LLMs) acquire substantial world knowledge during pre-training.<n>Post-training techniques such as supervised fine-tuning (SFT) shape this knowledge change behavior.<n>We evaluate closed-book question answering (CBQA) performance across five LLMs from the LLaMA-2 and LLaMA-3 families.
  arXiv  Detail & Related papers  (2025-09-20T09:40:32Z)
• EfficientLLM: Efficiency in Large Language Models [64.3537131208038]
  Large Language Models (LLMs) have driven significant progress, yet their growing counts and context windows incur prohibitive compute, energy, and monetary costs.<n>We introduce EfficientLLM, a novel benchmark and the first comprehensive empirical study evaluating efficiency techniques for LLMs at scale.
  arXiv  Detail & Related papers  (2025-05-20T02:27:08Z)
• Mitigating Image Captioning Hallucinations in Vision-Language Models [13.707454974844095]
  Hallucinations in vision-language models hinder reliability and real-world applicability.<n>We propose a novel test-time adaptation framework using reinforcement learning to mitigate hallucinations during inference.<n>Our approach outperforms state-of-the-art baselines with a 68.3% improvement in hallucination mitigation.
  arXiv  Detail & Related papers  (2025-05-06T10:55:21Z)
• DataComp-LM: In search of the next generation of training sets for language models [200.5293181577585]
  DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments with the goal of improving language models. We provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations. Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.
  arXiv  Detail & Related papers  (2024-06-17T17:42:57Z)
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning [61.10299147201369]
  This paper introduces a novel autonomous RL approach, called DigiRL, for training in-the-wild device control agents. We build a scalable and parallelizable Android learning environment equipped with a VLM-based evaluator. We demonstrate the effectiveness of DigiRL using the Android-in-the-Wild dataset, where our 1.3B VLM trained with RL achieves a 49.5% absolute improvement.
  arXiv  Detail & Related papers  (2024-06-14T17:49:55Z)
• SLYKLatent: A Learning Framework for Gaze Estimation Using Deep Facial Feature Learning [0.0]
  We present SLYKLatent, a novel approach for enhancing gaze estimation by addressing appearance instability challenges in datasets. SLYKLatent utilizes Self-Supervised Learning for initial training with facial expression datasets, followed by refinement with a patch-based tri-branch network. Our evaluation on benchmark datasets achieves a 10.9% improvement on Gaze360, supersedes top MPIIFaceGaze results with 3.8%, and leads on a subset of ETH-XGaze by 11.6%.
  arXiv  Detail & Related papers  (2024-02-02T16:47:18Z)
• Self-Supervised Pretraining Improves Performance and Inference Efficiency in Multiple Lung Ultrasound Interpretation Tasks [65.23740556896654]
  We investigated whether self-supervised pretraining could produce a neural network feature extractor applicable to multiple classification tasks in lung ultrasound analysis. When fine-tuning on three lung ultrasound tasks, pretrained models resulted in an improvement of the average across-task area under the receiver operating curve (AUC) by 0.032 and 0.061 on local and external test sets respectively.
  arXiv  Detail & Related papers  (2023-09-05T21:36:42Z)
• Speech separation with large-scale self-supervised learning [41.96634125460265]
  Self-supervised learning (SSL) methods such as WavLM have shown promising speech separation (SS) results in small-scale simulation-based experiments. We extend the exploration of the SSL-based SS by massively scaling up both the pre-training data (more than 300K hours) and fine-tuning data (10K hours)
  arXiv  Detail & Related papers  (2022-11-09T20:00:21Z)
• Improving the Training Recipe for a Robust Conformer-based Hybrid Model [46.78701739177677]
  We investigate various methods for speaker adaptive training (SAT) based on feature-space approaches for a conformer-based acoustic model (AM) We propose a method, called weighted-Simple-Add, which adds weighted speaker information vectors to the input of the multi-head self-attention module of the conformer AM. We extend and improve this recipe where we achieve 11% relative improvement in terms of word-error-rate (WER) on Switchboard 300h Hub5'00 dataset.
  arXiv  Detail & Related papers  (2022-06-26T20:01:08Z)
• Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems [63.713297451300086]
  We present results from a large-scale experiment on pretraining encoders with non-embedding parameter counts ranging from 700M to 9.3B. Their subsequent distillation into smaller models ranging from 17M-170M parameters, and their application to the Natural Language Understanding (NLU) component of a virtual assistant system.
  arXiv  Detail & Related papers  (2022-06-15T20:44:23Z)
• DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing [117.41016786835452]
  This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model. vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. We propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics.
  arXiv  Detail & Related papers  (2021-11-18T06:48:00Z)
• Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models [15.237992590162593]
  We present results from Alexa speech teams on semi-supervised learning (SSL) of acoustic models (AM) We discuss SSL for AMs in a small footprint setting, showing that a smaller capacity model trained with 1 million hours of unsupervised data can outperform a baseline supervised system by 14.3% word error rate reduction (WERR) We then switch to SSL using larger student models in low data regimes; while learning efficiency with unsupervised data is higher, student models may outperform teacher models in such a setting.
  arXiv  Detail & Related papers  (2021-06-11T02:23:40Z)
• Comparing CTC and LFMMI for out-of-domain adaptation of wav2vec 2.0 acoustic model [6.338178373376447]
  We investigate if the wav2vec 2.0 self-supervised pretraining helps mitigate the overfitting issues with connectionist temporal classification. For supervised adaptation of the wav2vec 2.0 model, both E2E-LFMMI and CTC achieve similar results. For Babel languages, we obtain relative improvements of 26% and 23% on Swahili (38h) and 18% and 17% on Tagalog (84h) respectively.
  arXiv  Detail & Related papers  (2021-04-06T14:56:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.