Articulation-Informed ASR: Integrating Articulatory Features into ASR via Auxiliary Speech Inversion and Cross-Attention Fusion

• URL: http://arxiv.org/abs/2510.08585v1
• Date: Wed, 01 Oct 2025 21:07:29 GMT
• Title: Articulation-Informed ASR: Integrating Articulatory Features into ASR via Auxiliary Speech Inversion and Cross-Attention Fusion
• Authors: Ahmed Adel Attia, Jing Liu, Carol Espy Wilson,
• Abstract summary: We revisit articulatory information in the era of deep learning.<n>We propose a framework that leverages articulatory representations both as an auxiliary task and as a pseudo-input to the recognition model.
• Score: 7.505518573248786
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prior works have investigated the use of articulatory features as complementary representations for automatic speech recognition (ASR), but their use was largely confined to shallow acoustic models. In this work, we revisit articulatory information in the era of deep learning and propose a framework that leverages articulatory representations both as an auxiliary task and as a pseudo-input to the recognition model. Specifically, we employ speech inversion as an auxiliary prediction task, and the predicted articulatory features are injected into the model as a query stream in a cross-attention module with acoustic embeddings as keys and values. Experiments on LibriSpeech demonstrate that our approach yields consistent improvements over strong transformer-based baselines, particularly under low-resource conditions. These findings suggest that articulatory features, once sidelined in ASR research, can provide meaningful benefits when reintroduced with modern architectures.

Related papers

• Training-Free Intelligibility-Guided Observation Addition for Noisy ASR [57.74127683005929]
  This paper proposes an intelligibility-guided observation addition (OA) method to improve speech recognition in noisy environments.<n>Experiments across diverse SE-ASR combinations and datasets demonstrate strong robustness and improvements over existing OA baselines.
  arXiv  Detail & Related papers  (2026-02-24T14:46:54Z)
• Beyond Transcription: Mechanistic Interpretability in ASR [26.551400592078213]
  Interpretability methods have recently gained significant attention, particularly in the context of large language models.<n>We adapt and apply established interpretability methods to examine how acoustic and semantic information evolves across layers in ASR systems.<n>Our experiments reveal previously unknown internal dynamics, including specific encoder-decoder interactions responsible for repetition hallucinations and semantic biases encoded deep within acoustic representations.
  arXiv  Detail & Related papers  (2025-08-21T15:42:53Z)
• AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation [113.75682363364004]
  AURORA is a framework designed to enhance genuine reasoning and language comprehension in reference audio-visual segmentation.<n>AURORA achieves state-of-the-art performance on Ref-AVS benchmarks and generalizes effectively to unreferenced segmentation.
  arXiv  Detail & Related papers  (2025-08-04T07:47:38Z)
• Echoes of Phonetics: Unveiling Relevant Acoustic Cues for ASR via Feature Attribution [19.32372029477596]
  We apply a feature attribution technique to identify the relevant acoustic cues for a modern Conformer-based ASR system.<n>By analyzing plosives, fricatives, and vowels, we assess how feature attributions align with their acoustic properties in the time and frequency domains.
  arXiv  Detail & Related papers  (2025-06-02T19:11:16Z)
• AV-RIR: Audio-Visual Room Impulse Response Estimation [49.469389715876915]
  Accurate estimation of Room Impulse Response (RIR) is important for speech processing and AR/VR applications. We propose AV-RIR, a novel multi-modal multi-task learning approach to accurately estimate the RIR from a given reverberant speech signal and visual cues of its corresponding environment.
  arXiv  Detail & Related papers  (2023-11-30T22:58:30Z)
• Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring [4.819085609772069]
  We propose a novel approach for enhancing contextual recognition within ASR systems via semantic lattice processing. Our solution consists of using Hidden Markov Models and Gaussian Mixture Models (HMM-GMM) along with Deep Neural Networks (DNN) models for better accuracy. We demonstrate the effectiveness of our proposed framework on the LibriSpeech dataset with empirical analyses.
  arXiv  Detail & Related papers  (2023-10-14T23:16:05Z)
• Exploring the Integration of Speech Separation and Recognition with Self-Supervised Learning Representation [83.36685075570232]
  This work provides an insightful investigation of speech separation in reverberant and noisy-reverberant scenarios as an ASR front-end. We explore multi-channel separation methods, mask-based beamforming and complex spectral mapping, as well as the best features to use in the ASR back-end model. A proposed integration using TF-GridNet-based complex spectral mapping and WavLM-based SSLR achieves a 2.5% word error rate in reverberant WHAMR! test set.
  arXiv  Detail & Related papers  (2023-07-23T05:39:39Z)
• Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning [25.743503223389784]
  We propose a reinforcement learning (RL) based framework called MSRL. We customize a reward function directly related to task-specific metrics. Experimental results on the LRS3 dataset show that the proposed method achieves state-of-the-art in both clean and various noisy conditions.
  arXiv  Detail & Related papers  (2022-12-10T14:01:54Z)
• Towards Disentangled Speech Representations [65.7834494783044]
  We construct a representation learning task based on joint modeling of ASR and TTS. We seek to learn a representation of audio that disentangles that part of the speech signal that is relevant to transcription from that part which is not. We show that enforcing these properties during training improves WER by 24.5% relative on average for our joint modeling task.
  arXiv  Detail & Related papers  (2022-08-28T10:03:55Z)
• End-to-End Active Speaker Detection [58.7097258722291]
  We propose an end-to-end training network where feature learning and contextual predictions are jointly learned. We also introduce intertemporal graph neural network (iGNN) blocks, which split the message passing according to the main sources of context in the ASD problem. Experiments show that the aggregated features from the iGNN blocks are more suitable for ASD, resulting in state-of-the art performance.
  arXiv  Detail & Related papers  (2022-03-27T08:55:28Z)
• ASR-Aware End-to-end Neural Diarization [15.172086811068962]
  We present a Conformer-based end-to-end neural diarization (EEND) model that uses both acoustic input and features derived from an automatic speech recognition (ASR) model. Three modifications to the Conformer-based EEND architecture are proposed to incorporate the features. Experiments on the two-speaker English conversations of Switchboard+SRE data sets show that multi-task learning with position-in-word information is the most effective way of utilizing ASR features.
  arXiv  Detail & Related papers  (2022-02-02T21:17:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.