Phone Segmentation and Recognition through Phonological Activation Mapping

• URL: http://arxiv.org/abs/2607.09020v1
• Date: Fri, 10 Jul 2026 01:05:30 GMT
• Title: Phone Segmentation and Recognition through Phonological Activation Mapping
• Abstract summary: Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately.<n>We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks.<n>We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality.
• Score: 76.99244720091627
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Phone segmentation and recognition are inherently related tasks, yet modern approaches typically model them separately. We argue that phonetic structure is already latent in the representations of self-supervised speech models (S3Ms), and one only needs to steer them to solve both tasks. We leverage S3M-based Phonological Activation Mapping (SPAM), which maps each S3M representation frame to a vector of phonological feature activations, such as voicing and nasality. On top of SPAM, we introduce two simple but effective lightweight, gradient-descent-free prediction heads: a recognition head and a segmentation head. Our method requires less than a minute of phonetic transcriptions, and generalizes to unseen phones during training. Across a diverse range of datasets, our approach attains strong segmentation and recognition performance.

Related papers

• Self-Supervised Speech Representations are More Phonetic than Semantic [52.02626675137819]
  Self-supervised speech models (S3Ms) have become an effective backbone for speech applications. We seek a more fine-grained analysis of the word-level linguistic properties encoded in S3Ms. Our study reveals that S3M representations consistently and significantly exhibit more phonetic than semantic similarity.
  arXiv  Detail & Related papers  (2024-06-12T20:04:44Z)
• Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation [79.05949524349005]
  We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from saliency maps. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps.
  arXiv  Detail & Related papers  (2024-03-02T10:03:21Z)
• Learning Speech Representation From Contrastive Token-Acoustic Pretraining [57.08426714676043]
  We propose "Contrastive Token-Acoustic Pretraining (CTAP)", which uses two encoders to bring phoneme and speech into a joint multimodal space. The proposed CTAP model is trained on 210k speech and phoneme pairs, achieving minimally-supervised TTS, VC, and ASR.
  arXiv  Detail & Related papers  (2023-09-01T12:35:43Z)
• Representation Learning With Hidden Unit Clustering For Low Resource Speech Applications [37.89857769906568]
  We describe an approach to self-supervised representation learning from raw audio using a hidden unit clustering (HUC) framework. The input to the model consists of audio samples that are windowed and processed with 1-D convolutional layers. The HUC framework, allowing the categorization of the representations into a small number of phoneme-like units, is used to train the model for learning semantically rich speech representations.
  arXiv  Detail & Related papers  (2023-07-14T13:02:10Z)
• What Do Self-Supervised Speech Models Know About Words? [23.163029143563893]
  Self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. Recent work has begun analyzing how S3Ms encode certain properties, such as phonetic and speaker information. We use lightweight analysis methods to study segment-level linguistic properties encoded in S3Ms.
  arXiv  Detail & Related papers  (2023-06-30T22:36:41Z)
• HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition [41.837538440839815]
  We propose a hierarchical cross-attention model (HCAM) approach to multi-modal emotion recognition. The input to the model consists of two modalities, i) audio data, processed through a learnable wav2vec approach and, ii) text data represented using a bidirectional encoder representations from transformers (BERT) model. In order to incorporate contextual knowledge and the information across the two modalities, the audio and text embeddings are combined using a co-attention layer.
  arXiv  Detail & Related papers  (2023-04-14T03:25:00Z)
• Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network [58.82343017711883]
  This paper investigates how to learn directly from unpaired phone sequences and speech utterances. GAN training is adopted in the first stage to find the mapping relationship between unpaired speech and phone sequence. In the second stage, another HMM model is introduced to train from the generator's output, which boosts the performance.
  arXiv  Detail & Related papers  (2022-07-29T09:29:28Z)
• Self-Supervised Speech Representation Learning: A Review [105.1545308184483]
  Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. This review presents approaches for self-supervised speech representation learning and their connection to other research areas.
  arXiv  Detail & Related papers  (2022-05-21T16:52:57Z)
• Unsupervised Acoustic Unit Discovery by Leveraging a Language-Independent Subword Discriminative Feature Representation [31.87235700253597]
  This paper tackles automatically discovering phone-like acoustic units (AUD) from unlabeled speech data. We propose a two-stage approach: the first stage learns a subword-discriminative feature representation and the second stage applies clustering to the learned representation and obtains phone-like clusters as the discovered acoustic units.
  arXiv  Detail & Related papers  (2021-04-02T11:43:07Z)
• Phoneme Boundary Detection using Learnable Segmental Features [31.203969460341817]
  Phoneme boundary detection plays an essential first step for a variety of speech processing applications. We propose a neural architecture coupled with a parameterized structured loss function to learn segmental representations for the task of phoneme boundary detection.
  arXiv  Detail & Related papers  (2020-02-11T14:03:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.