AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders

• URL: http://arxiv.org/abs/2602.05027v2
• Date: Fri, 06 Feb 2026 13:35:19 GMT
• Title: AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders
• Abstract summary: We train SAEs across all encoder layers of Whisper and HuBERT.<n>Over 50% of the features remain consistent across random seeds, and reconstruction quality is preserved.<n>We find SAE features correlated with human EEG activity during speech perception, indicating alignment with human neural processing.
• Score: 10.728025430620647
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse Autoencoders (SAEs) are powerful tools for interpreting neural representations, yet their use in audio remains underexplored. We train SAEs across all encoder layers of Whisper and HuBERT, provide an extensive evaluation of their stability, interpretability, and show their practical utility. Over 50% of the features remain consistent across random seeds, and reconstruction quality is preserved. SAE features capture general acoustic and semantic information as well as specific events, including environmental noises and paralinguistic sounds (e.g. laughter, whispering) and disentangle them effectively, requiring removal of only 19-27% of features to erase a concept. Feature steering reduces Whisper's false speech detections by 70% with negligible WER increase, demonstrating real-world applicability. Finally, we find SAE features correlated with human EEG activity during speech perception, indicating alignment with human neural processing. The code and checkpoints are available at https://github.com/audiosae/audiosae_demo.

Related papers

• Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders [7.378893412842889]
  We investigate whether hallucinations can be detected and mitigated through Whisper's internal representations.<n>We show that both spaces encode linearly separable hallucination-related information.<n>We propose two steering strategies: activation-space steering and SAE latent-space steering.
  arXiv  Detail & Related papers  (2026-06-05T17:26:23Z)
• Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition [58.25449304752214]
  We study whether explicit acoustic cues are grounded when the raw audio is already available.<n>We deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set.<n>Aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance.<n>We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.
  arXiv  Detail & Related papers  (2026-06-05T14:26:06Z)
• Learning Interpretable Features in Audio Latent Spaces via Sparse Autoencoders [4.757470067755357]
  We train SAEs on audio autoencoder latents, then learn linear mappings from SAE features to discretized acoustic properties.<n>This enables both controllable manipulation and analysis of the AI music generation process.
  arXiv  Detail & Related papers  (2025-10-27T19:35:39Z)
• Learning Robust Spatial Representations from Binaural Audio through Feature Distillation [64.36563387033921]
  We investigate the use of a pretraining stage based on feature distillation to learn a robust spatial representation of speech without the need for data labels.<n>Our experiments demonstrate that the pretrained models show improved performance in noisy and reverberant environments.
  arXiv  Detail & Related papers  (2025-08-28T15:43:15Z)
• Automatically Interpreting Millions of Features in Large Language Models [1.8035046415192353]
  sparse autoencoders (SAEs) can be used to transform activations into a higher-dimensional latent space.<n>We build an open-source pipeline to generate and evaluate natural language explanations for SAE features.<n>Our large-scale analysis confirms that SAE latents are indeed much more interpretable than neurons.
  arXiv  Detail & Related papers  (2024-10-17T17:56:01Z)
• Explaining Deep Learning Embeddings for Speech Emotion Recognition by Predicting Interpretable Acoustic Features [5.678610585849838]
  Pre-trained deep learning embeddings have consistently shown superior performance over handcrafted acoustic features in speech emotion recognition. Unlike acoustic features with clear physical meaning, these embeddings lack clear interpretability. This paper proposes a modified probing approach to explain deep learning embeddings in the speech emotion space.
  arXiv  Detail & Related papers  (2024-09-14T19:18:56Z)
• SafeEar: Content Privacy-Preserving Audio Deepfake Detection [17.859275594843965]
  We propose SafeEar, a novel framework that aims to detect deepfake audios without relying on accessing the speech content within. Our key idea is to devise a neural audio into a novel decoupling model that well separates the semantic and acoustic information from audio samples. In this way, no semantic content will be exposed to the detector.
  arXiv  Detail & Related papers  (2024-09-14T02:45:09Z)
• Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis [67.73554826428762]
  We propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech, enabling efficient synthesis of clean, realistic speech from noisy signals.
  arXiv  Detail & Related papers  (2022-03-31T17:57:10Z)
• CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition [91.33781557979819]
  We introduce a new dataset, Cantonese In-car Audio-Visual Speech Recognition (CI-AVSR) It consists of 4,984 samples (8.3 hours) of 200 in-car commands recorded by 30 native Cantonese speakers. We provide detailed statistics of both the clean and the augmented versions of our dataset.
  arXiv  Detail & Related papers  (2022-01-11T06:32:12Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• Learning Audio-Visual Dereverberation [87.52880019747435]
  Reverberation from audio reflecting off surfaces and objects in the environment not only degrades the quality of speech for human perception, but also severely impacts the accuracy of automatic speech recognition. Our idea is to learn to dereverberate speech from audio-visual observations. We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed sounds and visual scene.
  arXiv  Detail & Related papers  (2021-06-14T20:01:24Z)
• Multi-task self-supervised learning for Robust Speech Recognition [75.11748484288229]
  This paper proposes PASE+, an improved version of PASE for robust speech recognition in noisy and reverberant environments. We employ an online speech distortion module, that contaminates the input signals with a variety of random disturbances. We then propose a revised encoder that better learns short- and long-term speech dynamics with an efficient combination of recurrent and convolutional networks.
  arXiv  Detail & Related papers  (2020-01-25T00:24:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.