Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking

• URL: http://arxiv.org/abs/2103.09879v1
• Date: Wed, 17 Mar 2021 19:36:04 GMT
• Title: Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking
• Abstract summary: We advance self-supervised learning from permutations, by pre-training a model to reorder shuffled parts of the spectrogram of an audio signal. We show that inverting permutations is a meaningful pretext task for learning audio representations in an unsupervised fashion.
• Score: 35.644680012765356
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised pre-training using so-called "pretext" tasks has recently shown impressive performance across a wide range of modalities. In this work, we advance self-supervised learning from permutations, by pre-training a model to reorder shuffled parts of the spectrogram of an audio signal, to improve downstream classification performance. We make two main contributions. First, we overcome the main challenges of integrating permutation inversions into an end-to-end training scheme, using recent advances in differentiable ranking. This was heretofore sidestepped by casting the reordering task as classification, fundamentally reducing the space of permutations that can be exploited. Our experiments validate that learning from all possible permutations improves the quality of the pre-trained representations over using a limited, fixed set. Second, we show that inverting permutations is a meaningful pretext task for learning audio representations in an unsupervised fashion. In particular, we improve instrument classification and pitch estimation of musical notes by reordering spectrogram patches in the time-frequency space.

Related papers

• Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners [33.10721304996431]
  Self-supervised learning has driven substantial progress in audio representation learning.<n>We introduce NAPE, a self-supervised framework in which a causal Transformer predicts each next patch embedding of a spectrogram from the previous ones.<n>It achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results.
  arXiv  Detail & Related papers  (2026-08-20T10:16:07Z)
• BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations [9.030040918514464]
  BEST-RQ-2 is an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme.<n>A ViT context encoder processes only the unmasked spectrogram regions, and a lightweight predictor infers targets for the masked regions.<n>On the X-ARES and XARES-LLM benchmarks, BEST-RQ-2 consistently outperforms one-stage baselines in overall transfer.
  arXiv  Detail & Related papers  (2026-06-29T09:52:43Z)
• Learning Permutation from Structure Without Supervision [16.16926085760151]
  Learning from structure without supervision induces a non-uniform uncertainty.<n>Existing methods control this process using a single global temperature.<n>We introduce an entropy-adaptive formulation of Gumbel-Sinkhorn that locally modulates temperature based on assignment uncertainty.
  arXiv  Detail & Related papers  (2026-05-25T08:08:47Z)
• Learn to Rank: Visual Attribution by Learning Importance Ranking [58.69028273772474]
  We propose a new approach to generate visual attribution maps for computer vision models.<n>Our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement.<n>Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations.
  arXiv  Detail & Related papers  (2026-04-07T12:53:22Z)
• Masked Autoencoders as Universal Speech Enhancer [5.670678893351032]
  Masked autoencoder based universal speech enhancer is trained in a self-supervised manner.<n>We show that the proposed method achieves state-of-the-art performance for both in-domain and out-of-domain evaluation datasets.
  arXiv  Detail & Related papers  (2026-02-02T18:13:59Z)
• Self-Ensemble Post Learning for Noisy Domain Generalization [18.4218677759831]
  This paper explores how to make existing methods rework when meeting noise.<n>We find that the latent features inside the model have certain discriminative capabilities.<n>We propose the Self-Ensemble Post Learning approach to diversify features which can be leveraged.
  arXiv  Detail & Related papers  (2025-12-11T17:09:35Z)
• Incremental Sequence Classification with Temporal Consistency [9.65650774513798]
  We address the problem of incremental sequence classification, where predictions are updated as new elements in the sequence are revealed.<n>We leverage a temporal-consistency condition that successive predictions should satisfy to develop a novel loss function for training incremental sequence classifiers.<n>Our results show that models trained with our method are better able to distinguish promising generations from unpromising ones after observing only a few tokens.
  arXiv  Detail & Related papers  (2025-05-22T11:37:53Z)
• Enhancing Recommendation with Denoising Auxiliary Task [2.819369786209738]
  Due to the arbitrariness of user behavior, the presence of noise poses a challenge to predicting their next actions in recommender systems. We propose a novel self-supervised Auxiliary Task Joint Training (ATJT) method aimed at more accurately reweighting noisy sequences in recommender systems.
  arXiv  Detail & Related papers  (2024-09-25T22:26:29Z)
• Sortformer: Seamless Integration of Speaker Diarization and ASR by Bridging Timestamps and Tokens [27.08293218877395]
  We propose Sortformer, a novel neural model for speaker diarization. We introduce Sort Loss, which enables a diarization model to autonomously resolve permutation, with or without PIL. Code and trained models will be made publicly available via the NVIDIA NeMo framework.
  arXiv  Detail & Related papers  (2024-09-10T17:20:11Z)
• Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering [78.2927924732142]
  We propose speaker-invariant clustering (Spin) as a novel self-supervised learning method. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU.
  arXiv  Detail & Related papers  (2023-05-18T15:59:36Z)
• Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training [102.14558233502514]
  Masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance.
  arXiv  Detail & Related papers  (2022-06-21T06:08:30Z)
• Collar-aware Training for Streaming Speaker Change Detection in Broadcast Speech [0.0]
  We present a novel training method for speaker change detection models. The proposed method uses an objective function which encourages the model to predict a single positive label within a specified collar.
  arXiv  Detail & Related papers  (2022-05-14T15:35:43Z)
• Learning neural audio features without supervision [14.526173151915662]
  We study the feasibility of combining both approaches, i.e., pre-training learnable jointly with the main architecture for downstream classification. We uncover crucial differences in properties of learned components when used in a supervised and self-supervised setting.
  arXiv  Detail & Related papers  (2022-03-29T12:59:08Z)
• Discovering Non-monotonic Autoregressive Orderings with Variational Inference [67.27561153666211]
  We develop an unsupervised parallelizable learner that discovers high-quality generation orders purely from training data. We implement the encoder as a Transformer with non-causal attention that outputs permutations in one forward pass. Empirical results in language modeling tasks demonstrate that our method is context-aware and discovers orderings that are competitive with or even better than fixed orders.
  arXiv  Detail & Related papers  (2021-10-27T16:08:09Z)
• Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
  We propose textitPrototypical, which does not require fitting additional parameters given the embedding network. Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced. We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
  arXiv  Detail & Related papers  (2021-10-22T01:55:01Z)
• Class-incremental Learning using a Sequence of Partial Implicitly Regularized Classifiers [0.0]
  In class-incremental learning, the objective is to learn a number of classes sequentially without having access to the whole training data. Our experiments on CIFAR100 dataset show that the proposed method improves the performance over SOTA by a large margin.
  arXiv  Detail & Related papers  (2021-04-04T10:02:45Z)
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining [59.169836983883656]
  COCO-LM is a new self-supervised learning framework that pretrains Language Models by COrrecting challenging errors and COntrasting text sequences. COCO-LM employs an auxiliary language model to mask-and-predict tokens in original text sequences. Our analyses reveal that COCO-LM's advantages come from its challenging training signals, more contextualized token representations, and regularized sequence representations.
  arXiv  Detail & Related papers  (2021-02-16T22:24:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.