Multimodal Consistency-Guided Reference-Free Data Selection for ASR Accent Adaptation

• URL: http://arxiv.org/abs/2602.13263v1
• Date: Tue, 03 Feb 2026 21:35:58 GMT
• Title: Multimodal Consistency-Guided Reference-Free Data Selection for ASR Accent Adaptation
• Authors: Ligong Lei, Wenwen Lu, Xudong Pang, Zaokere Kadeer, Aishan Wumaier,
• Abstract summary: We introduce a multimodal consistency-guided, reference-free data selection pipeline for ASR accent adaptation.<n>The pipeline scores each hypothesis using two reference-free signals: speech-text alignment in a shared embedding space and predicted word error rate.<n>A simple percentile-based selection rule retains reliable pseudo-labels for fine-tuning while discarding noisy utterances.
• Score: 0.05219568203653524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic speech recognition (ASR) systems often degrade on accented speech because acoustic-phonetic and prosodic shifts induce a mismatch to training data, making labeled accent adaptation costly. However, common pseudo-label selection heuristics are largely text-centric (e.g., perplexity (PPL) filtering) and can prefer fluent yet acoustically mismatched hypotheses, leading to error amplification when fine-tuning. To address this, we introduce a multimodal consistency-guided, reference-free data selection pipeline for ASR accent adaptation under a transductive, label-free protocol. The pipeline starts with a target-aware preselection step based on submodular mutual information to improve query relevance and reduce downstream computation. It then generates multiple pseudo-transcriptions per utterance via perturbation-based decoding and scores each hypothesis using two reference-free signals: speech--text alignment in a shared embedding space and predicted word error rate (WER). A simple percentile-based selection rule retains reliable pseudo-labels for fine-tuning while discarding noisy utterances. In an in-domain setting, selecting ~1.5k utterances from a 30k pool achieves 10.91% WER, close to 10.45% obtained using 30k supervised labels. In a cross-domain setting with a mismatched candidate pool, consistency-filtered subsets avoid the degradation caused by unfiltered pseudo-labels under strong accent shift, and matched-hour experiments on a stronger ASR backbone further confirm gains over random sampling and recent selection baselines.

Related papers

• A Neural Model for Contextual Biasing Score Learning and Filtering [11.862176451777286]
  We use an attention-based biasing decoder to produce scores for candidate phrases based on acoustic information extracted by an ASR encoder.<n>We introduce a per-token discriminative objective that encourages higher scores for ground-truth phrases while suppressing distractors.<n>Our approach is modular and can be used with any ASR system, and the filtering mechanism can potentially boost performance of other biasing methods.
  arXiv  Detail & Related papers  (2025-10-27T20:41:52Z)
• MARS-Sep: Multimodal-Aligned Reinforced Sound Separation [72.85468563236005]
  MARS-Sep is a reinforcement learning framework for sound separation.<n>It learns a factorized Beta mask policy that is optimized by a clipped trust-region surrogate.<n>Experiments on multiple benchmarks demonstrate consistent gains in Text-, Audio-, and Image-Queried separation.
  arXiv  Detail & Related papers  (2025-10-12T09:05:28Z)
• Efficient Data Selection for Domain Adaptation of ASR Using Pseudo-Labels and Multi-Stage Filtering [11.50314008820538]
  Fine-tuning pretrained ASR models for specific domains is challenging for small organizations with limited labeled data and computational resources.<n>We propose a robust approach that improves ASR adaptation by filtering pseudo-labels generated using Whisper and Zipformer.
  arXiv  Detail & Related papers  (2025-06-04T08:11:24Z)
• Transcription-Free Fine-Tuning of Speech Separation Models for Noisy and Reverberant Multi-Speaker Automatic Speech Recognition [18.50957174600796]
  Solution to automatic speech recognition (ASR) of overlapping speakers is to separate speech and then perform ASR on the separated signals. Currently, the separator produces artefacts which often degrade ASR performance. This paper proposes a transcription-free method for joint training using only audio signals.
  arXiv  Detail & Related papers  (2024-06-13T08:20:58Z)
• Iterative pseudo-forced alignment by acoustic CTC loss for self-supervised ASR domain adaptation [80.12316877964558]
  High-quality data labeling from specific domains is costly and human time-consuming. We propose a self-supervised domain adaptation method, based upon an iterative pseudo-forced alignment algorithm.
  arXiv  Detail & Related papers  (2022-10-27T07:23:08Z)
• Speaker Embedding-aware Neural Diarization: a Novel Framework for Overlapped Speech Diarization in the Meeting Scenario [51.5031673695118]
  We reformulate overlapped speech diarization as a single-label prediction problem. We propose the speaker embedding-aware neural diarization (SEND) system.
  arXiv  Detail & Related papers  (2022-03-18T06:40:39Z)
• S3: Supervised Self-supervised Learning under Label Noise [53.02249460567745]
  In this paper we address the problem of classification in the presence of label noise. In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space. Our method significantly surpasses previous methods on both CIFARCIFAR100 with artificial noise and real-world noisy datasets such as WebVision and ANIMAL-10N.
  arXiv  Detail & Related papers  (2021-11-22T15:49:20Z)
• Single-channel speech separation using Soft-minimum Permutation Invariant Training [60.99112031408449]
  A long-lasting problem in supervised speech separation is finding the correct label for each separated speech signal. Permutation Invariant Training (PIT) has been shown to be a promising solution in handling the label ambiguity problem. In this work, we propose a probabilistic optimization framework to address the inefficiency of PIT in finding the best output-label assignment.
  arXiv  Detail & Related papers  (2021-11-16T17:25:05Z)
• Label-Synchronous Speech-to-Text Alignment for ASR Using Forward and Backward Transformers [49.403414751667135]
  This paper proposes a novel label-synchronous speech-to-text alignment technique for automatic speech recognition (ASR) The proposed method re-defines the speech-to-text alignment as a label-synchronous text mapping problem. Experiments using the corpus of spontaneous Japanese (CSJ) demonstrate that the proposed method provides an accurate utterance-wise alignment.
  arXiv  Detail & Related papers  (2021-04-21T03:05:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.