Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognition

• URL: http://arxiv.org/abs/2601.12436v1
• Date: Sun, 18 Jan 2026 14:46:08 GMT
• Title: Purification Before Fusion: Toward Mask-Free Speech Enhancement for Robust Audio-Visual Speech Recognition
• Authors: Linzhi Wu, Xingyu Zhang, Hao Yuan, Yakun Zhang, Changyan Zheng, Liang Xie, Tiejun Liu, Erwei Yin,
• Abstract summary: High-noise audio inputs are prone to introducing adverse interference into the feature fusion process.<n>We propose an end-to-end noise-robust AVSR framework coupled with speech enhancement.<n>Our method preserves speech semantic integrity to achieve robust recognition performance.
• Score: 13.50064027453736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-visual speech recognition (AVSR) typically improves recognition accuracy in noisy environments by integrating noise-immune visual cues with audio signals. Nevertheless, high-noise audio inputs are prone to introducing adverse interference into the feature fusion process. To mitigate this, recent AVSR methods often adopt mask-based strategies to filter audio noise during feature interaction and fusion, yet such methods risk discarding semantically relevant information alongside noise. In this work, we propose an end-to-end noise-robust AVSR framework coupled with speech enhancement, eliminating the need for explicit noise mask generation. This framework leverages a Conformer-based bottleneck fusion module to implicitly refine noisy audio features with video assistance. By reducing modality redundancy and enhancing inter-modal interactions, our method preserves speech semantic integrity to achieve robust recognition performance. Experimental evaluations on the public LRS3 benchmark suggest that our method outperforms prior advanced mask-based baselines under noisy conditions.

Related papers

• Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion [46.072071890391356]
  We propose a novel framework that adaptively reweights audio and visual features based on token-level acoustic corruption scores.<n>Using an audio-visual feature fusion-based router, our method down-weights unreliable audio tokens and reinforces visual cues through cross-attention gated in each decoder layer.<n> Experiments on LRS3 demonstrate that our approach achieves an 16.51-42.67% relative reduction in word error rate compared to AV-HuBERT.
  arXiv  Detail & Related papers  (2025-08-26T07:05:48Z)
• Cocktail-Party Audio-Visual Speech Recognition [58.222892601847924]
  This study introduces a novel audio-visual cocktail-party dataset designed to benchmark current AVSR systems.<n>We contribute a 1526-hour AVSR dataset comprising both talking-face and silent-face segments, enabling significant performance gains in cocktail-party environments.<n>Our approach reduces WER by 67% relative to the state-of-the-art, reducing WER from 119% to 39.2% in extreme noise, without relying on explicit segmentation cues.
  arXiv  Detail & Related papers  (2025-06-02T19:07:51Z)
• Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberation [13.575063025878208]
  We introduce a new paradigm for active sound modification: Active Speech Enhancement (ASE)<n>We propose a novel Transformer-Mamba-based architecture, along with a task-specific loss function designed to jointly optimize interference suppression and signal enrichment.<n>Our method outperforms existing baselines across multiple speech processing tasks -- including denoising, dereverberation, and declipping.
  arXiv  Detail & Related papers  (2025-05-22T17:10:18Z)
• AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection [57.649223695021114]
  We present a novel weakly supervised framework that leverages audio-visual collaboration for robust video anomaly detection.<n>Our framework demonstrates superior performance across multiple benchmarks, with audio integration significantly boosting anomaly detection accuracy.
  arXiv  Detail & Related papers  (2025-04-06T13:59:16Z)
• Inference and Denoise: Causal Inference-based Neural Speech Enhancement [83.4641575757706]
  This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. The proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE.
  arXiv  Detail & Related papers  (2022-11-02T15:03:50Z)
• Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition [29.05833230733178]
  We propose Visual Context-driven Audio Feature Enhancement module (V-CAFE) to enhance the input noisy audio speech with a help of audio-visual correspondence. The proposed V-CAFE is designed to capture the transition of lip movements, namely visual context and to generate a noise reduction mask by considering the obtained visual context. The effectiveness of the proposed method is evaluated in noisy speech recognition and overlapped speech recognition experiments using the two largest audio-visual datasets, LRS2 and LRS3.
  arXiv  Detail & Related papers  (2022-07-13T08:07:19Z)
• Robust Semantic Communications with Masked VQ-VAE Enabled Codebook [56.63571713657059]
  We propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise. To combat the semantic noise, the adversarial training with weight is developed to incorporate the samples with semantic noise in the training dataset. We develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features.
  arXiv  Detail & Related papers  (2022-06-08T16:58:47Z)
• Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction [109.44933866397123]
  Noise robustness is essential for deploying automatic speech recognition systems in real-world environments. We employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
  arXiv  Detail & Related papers  (2021-10-28T20:39:02Z)
• Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognition [64.9317368575585]
  This paper proposes a gated recurrent fusion (GRF) method with joint training framework for robust end-to-end ASR. The GRF algorithm is used to dynamically combine the noisy and enhanced features. The proposed method achieves the relative character error rate (CER) reduction of 10.04% over the conventional joint enhancement and transformer method.
  arXiv  Detail & Related papers  (2020-11-09T08:52:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.