Mechanistic Interpretability of ASR models using Sparse Autoencoders

• URL: http://arxiv.org/abs/2605.12225v1
• Date: Tue, 12 May 2026 15:02:20 GMT
• Title: Mechanistic Interpretability of ASR models using Sparse Autoencoders
• Abstract summary: Internal mechanisms of deep Transformer-based NLP models remain largely a mystery.<n>Sparse Autoencoders (SAE) have emerged to understand these mechanisms by projecting dense representations into a sparse vector.<n>In this work, a SAE is applied to Whisper, a Transformer-based ASR, training a high-dimensional sparse latent space on frame-level embeddings extracted from the Whisper encoder.
• Score: 2.835219195516112
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Understanding the internal machinations of deep Transformer-based NLP models is more crucial than ever as these models see widespread use in various domains that affect the public at large, such as industry, academia, finance, health. While these models have advanced rapidly, their internal mechanisms remain largely a mystery. Techniques such as Sparse Autoencoders (SAE) have emerged to understand these mechanisms by projecting dense representations into a sparse vector. While existing research has demonstrated the viability of the SAE in interpreting text-based Large Language Models (LLMs), there are no equivalent studies that demonstrate the application of a SAE to audio processing models like Automatic Speech Recognizers (ASRs). In this work, a SAE is applied to Whisper, a Transformer-based ASR, training a high-dimensional sparse latent space on frame-level embeddings extracted from the Whisper encoder. Our work uncovers diverse monosemantic features across linguistic and non-linguistic boundaries, and demonstrates cross-lingual feature steering. This work establishes the viability of a SAE model and demonstrates that Whisper encodes a rich amount of linguistic information.

Related papers

• Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models [80.45129499188461]
  We introduce Qwen-Scope, an open-source suite of sparse autoencoders (SAEs) built on the Qwen model family.<n>We show that SAEs can go beyond post-hoc analysis to serve as practical interfaces for model development along four directions.
  arXiv  Detail & Related papers  (2026-05-12T10:01:06Z)
• Learning Retrieval Models with Sparse Autoencoders [9.69714089726174]
  SPLARE is a method to train SAE-based LSR models.<n>We show that SPLARE consistently outperforms vocabulary-based LSR in multilingual and out-of-domain settings.
  arXiv  Detail & Related papers  (2026-02-27T10:50:54Z)
• ProtSAE: Disentangling and Interpreting Protein Language Models via Semantically-Guided Sparse Autoencoders [30.219733023958188]
  Sparse Autoencoder (SAE) has emerged as a powerful tool for mechanistic interpretability of large language models.<n>We propose a semantically-guided SAE, called ProtSAE.<n>We show that ProtSAE learns more biologically relevant and interpretable hidden features compared to previous methods.
  arXiv  Detail & Related papers  (2025-08-26T11:20:31Z)
• Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models [50.587868616659826]
  We introduce a comprehensive framework for evaluating monosemanticity at the neuron-level in vision representations.<n>Our experimental results reveal that SAEs trained on Vision-Language Models significantly enhance the monosemanticity of individual neurons.
  arXiv  Detail & Related papers  (2025-04-03T17:58:35Z)
• Sparse Autoencoder Insights on Voice Embeddings [3.2377830280631468]
  This study applies sparse autoencoders to speaker embeddings generated from a Titanet model.<n>The extracted features exhibit characteristics similar to those found in Large Language Model embeddings, including feature splitting and steering.<n>The analysis reveals that the autoencoder can identify and manipulate features such as language and music, which are not evident in the original embedding.
  arXiv  Detail & Related papers  (2025-01-31T19:21:43Z)
• MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models [59.80042864360884]
  Speaker-attributed automatic speech recognition (SA-ASR) aims to transcribe speech while assigning transcripts to the corresponding speakers accurately.<n>This paper introduces a novel approach, leveraging a frozen multilingual ASR model to incorporate speaker attribution into the transcriptions.
  arXiv  Detail & Related papers  (2024-11-27T09:01:08Z)
• Disentangling Dense Embeddings with Sparse Autoencoders [0.0]
  Sparse autoencoders (SAEs) have shown promise in extracting interpretable features from complex neural networks. We present one of the first applications of SAEs to dense text embeddings from large language models. We show that the resulting sparse representations maintain semantic fidelity while offering interpretability.
  arXiv  Detail & Related papers  (2024-08-01T15:46:22Z)
• Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations [16.577870835480585]
  We present a comprehensive analysis on building ASR systems with discrete codes. We investigate different methods for training such as quantization schemes and time-domain vs spectral feature encodings. We introduce a pipeline that outperforms Encodec at similar bit-rate.
  arXiv  Detail & Related papers  (2024-07-03T20:51:41Z)
• LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models [50.259006481656094]
  We present a novel interactive application aimed towards understanding the internal mechanisms of large vision-language models. Our interface is designed to enhance the interpretability of the image patches, which are instrumental in generating an answer. We present a case study of how our application can aid in understanding failure mechanisms in a popular large multi-modal model: LLaVA.
  arXiv  Detail & Related papers  (2024-04-03T23:57:34Z)
• On decoder-only architecture for speech-to-text and large language model integration [59.49886892602309]
  Speech-LLaMA is a novel approach that effectively incorporates acoustic information into text-based large language models. We conduct experiments on multilingual speech-to-text translation tasks and demonstrate a significant improvement over strong baselines.
  arXiv  Detail & Related papers  (2023-07-08T06:47:58Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.