Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition

• URL: http://arxiv.org/abs/2602.19316v1
• Date: Sun, 22 Feb 2026 19:38:21 GMT
• Title: Pay Attention to CTC: Fast and Robust Pseudo-Labelling for Unified Speech Recognition
• Authors: Alexandros Haliassos, Rodrigo Mira, Stavros Petridis,
• Abstract summary: Unified Speech Recognition has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition.<n>We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets.<n>Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously.
• Score: 61.39209522608919
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unified Speech Recognition (USR) has emerged as a semi-supervised framework for training a single model for audio, visual, and audiovisual speech recognition, achieving state-of-the-art results on in-distribution benchmarks. However, its reliance on autoregressive pseudo-labelling makes training expensive, while its decoupled supervision of CTC and attention branches increases susceptibility to self-reinforcing errors, particularly under distribution shifts involving longer sequences, noise, or unseen domains. We propose CTC-driven teacher forcing, where greedily decoded CTC pseudo-labels are fed into the decoder to generate attention targets in a single forward pass. Although these can be globally incoherent, in the pseudo-labelling setting they enable efficient and effective knowledge transfer. Because CTC and CTC-driven attention pseudo-labels have the same length, the decoder can predict both simultaneously, benefiting from the robustness of CTC and the expressiveness of attention without costly beam search. We further propose mixed sampling to mitigate the exposure bias of the decoder relying solely on CTC inputs. The resulting method, USR 2.0, halves training time, improves robustness to out-of-distribution inputs, and achieves state-of-the-art results on LRS3, LRS2, and WildVSR, surpassing USR and modality-specific self-supervised baselines.

Related papers

• CR-CTC: Consistency regularization on CTC for improved speech recognition [18.996929774821822]
  Connectionist Temporal Classification (CTC) is a widely used method for automatic speech recognition (ASR)<n>We propose the Consistency-Regularized CTC (CR-CTC), which enforces consistency between two CTC distributions.<n>It significantly improves the CTC performance, achieving state-of-the-art results comparable to those attained by transducer or systems combining CTC and attention-based encoder-decoder (CTC/AED)
  arXiv  Detail & Related papers  (2024-10-07T14:56:07Z)
• Audio-Visual Efficient Conformer for Robust Speech Recognition [91.3755431537592]
  We propose to improve the noise of the recently proposed Efficient Conformer Connectionist Temporal Classification architecture by processing both audio and visual modalities. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps.
  arXiv  Detail & Related papers  (2023-01-04T05:36:56Z)
• Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training [102.14558233502514]
  Masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance.
  arXiv  Detail & Related papers  (2022-06-21T06:08:30Z)
• Cross-Modal Attention Consistency for Video-Audio Unsupervised Learning [141.38505371646482]
  Cross-modal correlation provides an inherent supervision for video unsupervised representation learning. This paper introduces a pretext task, Cross-Modal Attention Consistency (CMAC), for exploring the bidirectional local correspondence property. CMAC aims to align the regional attention generated purely from the visual signal with the target attention generated under the guidance of acoustic signal.
  arXiv  Detail & Related papers  (2021-06-13T07:41:15Z)
• Alignment Knowledge Distillation for Online Streaming Attention-based Speech Recognition [46.69852287267763]
  This article describes an efficient training method for online streaming attention-based encoder-decoder (AED) automatic speech recognition (ASR) systems. The proposed method significantly reduces recognition errors and emission latency simultaneously. The best MoChA system shows performance comparable to that of RNN-transducer (RNN-T)
  arXiv  Detail & Related papers  (2021-02-28T08:17:38Z)
• Intermediate Loss Regularization for CTC-based Speech Recognition [58.33721897180646]
  We present a simple and efficient auxiliary loss function for automatic speech recognition (ASR) based on the connectionist temporal classification ( CTC) objective. We evaluate the proposed method on various corpora, reaching word error rate (WER) 9.9% on the WSJ corpus and character error rate (CER) 5.2% on the AISHELL-1 corpus respectively.
  arXiv  Detail & Related papers  (2021-02-05T15:01:03Z)
• Focus on the present: a regularization method for the ASR source-target attention layer [45.73441417132897]
  This paper introduces a novel method to diagnose the source-target attention in state-of-the-art end-to-end speech recognition models. Our method is based on the fact that both, CTC and source-target attention, are acting on the same encoder representations. We found that the source-target attention heads are able to predict several tokens ahead of the current one.
  arXiv  Detail & Related papers  (2020-11-02T18:56:33Z)
• CTC-synchronous Training for Monotonic Attention Model [43.0382262234792]
  backward probabilities cannot be leveraged in the alignment process during training due to left-to-right dependency in the decoder. We propose CTC-synchronous training ( CTC-ST), in which MoChA uses CTC alignments to learn optimal monotonic alignments. The entire model is jointly optimized so that the expected boundaries from MoChA are synchronized with the alignments.
  arXiv  Detail & Related papers  (2020-05-10T16:48:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.