Time-Frequency Weighted Losses for Phoneme Reconstruction in DNN-Based Speech Enhancement

• URL: http://arxiv.org/abs/2606.21635v1
• Date: Fri, 19 Jun 2026 17:38:03 GMT
• Title: Time-Frequency Weighted Losses for Phoneme Reconstruction in DNN-Based Speech Enhancement
• Abstract summary: We propose a TF weighting framework that modulates the SDR objective based on local speech presence, speech-to-interference ratio (SIR), and spectral flux.<n> Experimental results show that our approach improves objective frequency-weighted enhancement metrics, as well as phoneme recognition accuracy.
• Score: 22.777861592418777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conventional training losses for speech enhancement based on the signal-to-distortion ratio (SDR) treat all time-frequency (TF) regions uniformly, overlooking the fine-grained spectral cues that are relevant to specific phoneme intelligibility. We propose a TF weighting framework that modulates the SDR objective based on local speech presence, speech-to-interference ratio (SIR), and spectral flux. By integrating these factors into a differentiable objective, the framework emphasizes TF bins with high speech-noise competition while also accounting for transient cues such as consonant bursts. Experimental results show that our approach improves objective frequency-weighted enhancement metrics, as well as phoneme recognition accuracy, particularly for consonants. Spectral analysis shows better reconstruction of mid-frequency structures at less adverse SIR.

Related papers

• SRENet: Spectral Re-Entry Network for Point Cloud Action Recognition [49.508770157706486]
  We propose SRENet to learn both global context and fine-grained temporal dynamics of motion from a frequency perspective for action recognition.<n>SRENet achieves state-of-the-art performance, validating the effectiveness of frequency modeling in point cloud-based action understanding.
  arXiv  Detail & Related papers  (2026-06-02T05:11:47Z)
• Spectral Gating Networks [65.9496901693099]
  We introduce Spectral Gating Networks (SGN) to introduce frequency-rich expressivity in feed-forward networks.<n>SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior.<n>It consistently improves accuracy-efficiency trade-offs under comparable computational budgets.
  arXiv  Detail & Related papers  (2026-02-07T20:00:49Z)
• Iterative Inference-time Scaling with Adaptive Frequency Steering for Image Super-Resolution [75.3690742776891]
  We propose Iterative Diffusion Inference-Time Scaling with Adaptive Frequency Steering (IAFS)<n>IAFS addresses the challenge of balancing perceptual quality and structural fidelity by progressively refining the generated image through iterative correction of structural deviations.<n>Experiments show that IAFS effectively resolves the perception-fidelity conflict, yielding consistently improved perceptual detail and structural accuracy, and outperforming existing inference-time scaling methods.
  arXiv  Detail & Related papers  (2025-12-29T15:09:20Z)
• Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation [13.886659472425393]
  Multimodal recommendation aims to integrate collaborative signals with heterogeneous content such as visual and textual information.<n>These issues are often exacerbated by naive fusion or shallow modeling strategies, leading to degraded generalization and poor robustness.<n>We propose a Structured Spectral Reasoning framework for frequency-aware multimodal recommendation.
  arXiv  Detail & Related papers  (2025-12-01T07:39:28Z)
• Frequency-Weighted Training Losses for Phoneme-Level DNN-based Speech Enhancement [15.332506773218315]
  We propose perceptually-informed variants of the SDR loss, formulated in the time-frequency domain and modulated by frequency-dependent weighting schemes.<n>We train the FaSNet multichannel speech enhancement model using these various losses.<n> Experimental results show that while standard metrics such as the SDR are only marginally improved, their perceptual frequency-weighted counterparts exhibit a more substantial improvement.
  arXiv  Detail & Related papers  (2025-06-23T14:52:34Z)
• mdctGAN: Taming transformer-based GAN for speech super-resolution with Modified DCT spectra [4.721572768262729]
  Speech super-resolution (SSR) aims to recover a high resolution (HR) speech from its corresponding low resolution (LR) counterpart. Recent SSR methods focus more on the reconstruction of the magnitude spectrogram, ignoring the importance of phase reconstruction. We propose mdctGAN, a novel SSR framework based on modified discrete cosine transform (MDCT)
  arXiv  Detail & Related papers  (2023-05-18T16:49:46Z)
• Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition [65.25325641528701]
  Motivated by the spectro-temporal level differences between disordered and normal speech that systematically manifest in articulatory imprecision, decreased volume and clarity, slower speaking rates and increased dysfluencies, novel spectro-temporal subspace basis embedding deep features derived by SVD decomposition of speech spectrum are proposed. Experiments conducted on the UASpeech corpus suggest the proposed spectro-temporal deep feature adapted systems consistently outperformed baseline i- adaptation by up to 263% absolute (8.6% relative) reduction in word error rate (WER) with or without data augmentation.
  arXiv  Detail & Related papers  (2022-01-14T16:56:43Z)
• Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation [73.1652905564163]
  We address the problem of separating individual speech signals from videos using audio-visual neural processing. Most conventional approaches utilize frame-wise matching criteria to extract shared information between co-occurring audio and video. We propose a cross-modal affinity network (CaffNet) that learns global correspondence as well as locally-varying affinities between audio and visual streams.
  arXiv  Detail & Related papers  (2021-03-25T15:39:12Z)
• Improving Stability of LS-GANs for Audio and Speech Signals [70.15099665710336]
  We show that encoding departure from normality computed in this vector space into the generator optimization formulation helps to craft more comprehensive spectrograms. We demonstrate the effectiveness of binding this metric for enhancing stability in training with less mode collapse compared to baseline GANs.
  arXiv  Detail & Related papers  (2020-08-12T17:41:25Z)
• Unsupervised Cross-Domain Speech-to-Speech Conversion with Time-Frequency Consistency [14.062850439230111]
  We propose a condition encouraging spectrogram consistency during the adversarial training procedure. Our experimental results on the Librispeech corpus show that the model trained with the TF consistency provides a perceptually better quality of speech-to-speech conversion.
  arXiv  Detail & Related papers  (2020-05-15T22:27:07Z)
• End-to-End Multi-Task Denoising for joint SDR and PESQ Optimization [43.15288441772729]
  Denoising networks learn mapping from noisy speech to clean one directly. Existing schemes have either of two critical issues: spectrum and metric mismatches. This paper presents a new end-to-end denoising framework with the goal of joint SDR and PESQ optimization.
  arXiv  Detail & Related papers  (2019-01-26T02:48:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.