LISE : Listenable Interpretable Speaker Embeddings

• URL: http://arxiv.org/abs/2606.21305v1
• Date: Fri, 19 Jun 2026 10:37:09 GMT
• Title: LISE : Listenable Interpretable Speaker Embeddings
• Abstract summary: Listenable Interpretable Speaker Embeddings (LISE) is a label-free framework that decomposes pretrained speaker embeddings into a small set of components.<n>This decomposition yields a structured representation that supports the analysis of what information has been encoded by speaker embeddings.<n>LISE preserves ASV performance with negligible EER degradation on x-vector and ECAPA-TDNN.
• Score: 15.561607569242605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural network-based automatic speaker verification (ASV) systems achieve impressive performance but their embedding representations remain opaque, lacking a structured and perceptually verifiable explanation of the vocal characteristics they encode. Existing approaches either require annotation of speaker attributes or introduce alternative representations whose interpretability is unvalidated with listeners. We propose Listenable Interpretable Speaker Embeddings (LISE), a label-free framework that decomposes pretrained speaker embeddings into a small set of components. This decomposition yields a structured representation that supports the analysis of what information has been encoded by speaker embeddings. LISE preserves ASV performance with negligible EER degradation on x-vector and ECAPA-TDNN. Crucially, the interpretability of these components for human listeners is demonstrated through listening experiments, where participants distinguished speakers with 83.9% accuracy.

Related papers

• Acoustic Cue Alignment in Audio Language Models for Speech Emotion Recognition [58.25449304752214]
  We study whether explicit acoustic cues are grounded when the raw audio is already available.<n>We deriving six interpretable acoustic concept tokens from the standardised eGeMAPS paralinguistic feature set.<n>Aligned tokens improve unweighted average recall (UAR), whereas shuffled, conflicting, or corrupted tokens reduce performance.<n>We argue that token-only interventions provide a practical way to probe audio-grounded cue use, robustness, and interpretability in ALM-based affective computing.
  arXiv  Detail & Related papers  (2026-06-05T14:26:06Z)
• SNAP: Speaker Nulling for Artifact Projection in Speech Deepfake Detection [51.096014381455454]
  Self-supervised learning-based speech encoders struggle to generalize across unseen speakers.<n>Speaker entanglement causes detectors to exploit speaker-specific correlations rather than artifact-related cues.<n>We introduce SNAP, a speaker-nulling framework, to mitigate this reliance.
  arXiv  Detail & Related papers  (2026-03-21T07:05:30Z)
• Koopman Regularized Deep Speech Disentanglement for Speaker Verification [6.659299099827954]
  Deep Koopman Speech Disentanglement Autoencoder (DKSD-AE)<n>We propose a structured autoencoder that combines a novel multi-step Koopman operator learning module with instance normalization to disentangle speaker and content dynamics.<n>Our findings suggest that Koopman-based temporal modelling, when combined with instance normalization, provides an efficient and principled solution for speaker-focused representation learning.
  arXiv  Detail & Related papers  (2026-03-05T17:30:18Z)
• Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers [50.9040167152168]
  We analyze neurons associated with k-means clusters of self-supervised features and i-vectors.<n>Our analysis reveals that these clusters correspond to broad phonetic and gender classes.<n>By protecting these neurons during pruning, we can significantly preserve performance on speaker-related task.
  arXiv  Detail & Related papers  (2025-06-26T18:54:26Z)
• Phonetic-aware speaker embedding for far-field speaker verification [25.50311094643337]
  We propose a joint-training speech recognition and speaker recognition framework to exploit phonetic content for far-field speaker verification. The framework encourages speaker embeddings to preserve phonetic information by matching the frame-based feature maps of a speaker embedding network with wav2vec's vectors. Results show that the proposed framework outperforms the standard speaker embedding on the VOiCES Challenge 2019 evaluation set and the VoxCeleb1 test set.
  arXiv  Detail & Related papers  (2023-11-27T08:45:35Z)
• Residual Information in Deep Speaker Embedding Architectures [4.619541348328938]
  This paper introduces an analysis over six sets of speaker embeddings extracted with some of the most recent and high-performing DNN architectures. The dataset includes 46 speakers uttering the same set of prompts, recorded in either a professional studio or their home environments. The results show that the discriminative power of the analyzed embeddings is very high, yet across all the analyzed architectures, residual information is still present in the representations.
  arXiv  Detail & Related papers  (2023-02-06T12:37:57Z)
• Content-Context Factorized Representations for Automated Speech Recognition [12.618527387900079]
  We introduce an unsupervised, encoder-agnostic method for factoring speech-encoder representations into explicit content-encoding representations and spurious context-encoding representations. We demonstrate improved performance on standard ASR benchmarks, as well as improved performance in both real-world and artificially noisy ASR scenarios.
  arXiv  Detail & Related papers  (2022-05-19T21:34:40Z)
• PL-EESR: Perceptual Loss Based END-TO-END Robust Speaker Representation Extraction [90.55375210094995]
  Speech enhancement aims to improve the perceptual quality of the speech signal by suppression of the background noise. We propose an end-to-end deep learning framework, dubbed PL-EESR, for robust speaker representation extraction.
  arXiv  Detail & Related papers  (2021-10-03T07:05:29Z)
• Content-Aware Speaker Embeddings for Speaker Diarisation [3.6398652091809987]
  The content-aware speaker embeddings (CASE) approach is proposed. Case factorises automatic speech recognition (ASR) from speaker recognition to focus on modelling speaker characteristics. Case achieved a 17.8% relative speaker error rate reduction over conventional methods.
  arXiv  Detail & Related papers  (2021-02-12T12:02:03Z)
• Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning [58.14807331265752]
  We show that better speaker embeddings can be learned by momentum contrastive learning. We generalize the self-supervised framework to a semi-supervised scenario where only a small portion of the data is labeled.
  arXiv  Detail & Related papers  (2020-12-13T23:23:39Z)
• Speaker Diarization with Lexical Information [59.983797884955]
  This work presents a novel approach for speaker diarization to leverage lexical information provided by automatic speech recognition. We propose a speaker diarization system that can incorporate word-level speaker turn probabilities with speaker embeddings into a speaker clustering process to improve the overall diarization accuracy.
  arXiv  Detail & Related papers  (2020-04-13T17:16:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.