Perceptually Aligning Representations of Music via Noise-Augmented Autoencoders

• URL: http://arxiv.org/abs/2511.05350v2
• Date: Mon, 10 Nov 2025 14:11:02 GMT
• Title: Perceptually Aligning Representations of Music via Noise-Augmented Autoencoders
• Authors: Mathias Rose Bjare, Giorgia Cantisani, Marco Pasini, Stefan Lattner, Gerhard Widmer,
• Abstract summary: We show that, after training an audio autoencoder in this manner, perceptually salient information is captured in coarser representation structures than with conventional training.<n>We show that such perceptual hierarchies improve latent diffusion decoding in the context of estimating surprisal in music pitches and predicting EEG-brain responses to music listening.
• Score: 13.596509137642103
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We argue that training autoencoders to reconstruct inputs from noised versions of their encodings, when combined with perceptual losses, yields encodings that are structured according to a perceptual hierarchy. We demonstrate the emergence of this hierarchical structure by showing that, after training an audio autoencoder in this manner, perceptually salient information is captured in coarser representation structures than with conventional training. Furthermore, we show that such perceptual hierarchies improve latent diffusion decoding in the context of estimating surprisal in music pitches and predicting EEG-brain responses to music listening. Pretrained weights are available on github.com/CPJKU/pa-audioic.

Related papers

• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• Adapting Neural Audio Codecs to EEG [27.20793132729464]
  We show that pretrained neural audio codecs can serve as effective starting points for EEG compression.<n>We propose DAC-MC, a multi-channel extension with attention-based cross-channel aggregation and channel-specific decoding.<n> Evaluations on the TUH Abnormal and Epilepsy datasets show that the adapted codecs preserve clinically relevant information.
  arXiv  Detail & Related papers  (2025-11-28T12:47:05Z)
• Unified Multimodal Model as Auto-Encoder [69.38946823657592]
  We introduce a paradigm regarding understanding as the encoder (I2T) that compresses images into text, and generation as the decoder (T2I) that reconstructs images from that text.<n>Our empirical results suggest that understanding can largely enhance generation (verified on GenEval), while generation, in turn, notably strengthens fine-grained visual perception.
  arXiv  Detail & Related papers  (2025-09-11T17:57:59Z)
• Epsilon-VAE: Denoising as Visual Decoding [61.29255979767292]
  We propose denoising as decoding, shifting from single-step reconstruction to iterative refinement.<n>Specifically, we replace the decoder with a diffusion process that iteratively refines noise to recover the original image.<n>By adopting iterative reconstruction through diffusion, our autoencoder, namely Epsilon-VAE, achieves high reconstruction quality.
  arXiv  Detail & Related papers  (2024-10-05T08:27:53Z)
• Challenging Decoder helps in Masked Auto-Encoder Pre-training for Dense Passage Retrieval [10.905033385938982]
  Masked auto-encoder (MAE) pre-training architecture has emerged as the most promising. We propose a novel token importance aware masking strategy based on pointwise mutual information to intensify the challenge of the decoder.
  arXiv  Detail & Related papers  (2023-05-22T16:27:10Z)
• End-to-End Binaural Speech Synthesis [71.1869877389535]
  We present an end-to-end speech synthesis system that combines a low-bitrate audio system with a powerful decoder. We demonstrate the capability of the adversarial loss in capturing environment effects needed to create an authentic auditory scene.
  arXiv  Detail & Related papers  (2022-07-08T05:18:36Z)
• Automatic Audio Captioning using Attention weighted Event based Embeddings [25.258177951665594]
  We propose an encoder-decoder architecture with light-weight (i.e. with lesser learnable parameters) Bi-LSTM recurrent layers for AAC. Our results show that an efficient AED based embedding extractor combined with temporal attention and augmentation techniques is able to surpass existing literature.
  arXiv  Detail & Related papers  (2022-01-28T05:54:19Z)
• Voice Activity Detection for Transient Noisy Environment Based on Diffusion Nets [13.558688470594674]
  We address voice activity detection in acoustic environments of transients and stationary noises. We exploit unique spatial patterns of speech and non-speech audio frames by independently learning their underlying geometric structure. A deep neural network is trained to separate speech from non-speech frames.
  arXiv  Detail & Related papers  (2021-06-25T17:05:26Z)
• Diffusion-Based Representation Learning [65.55681678004038]
  We augment the denoising score matching framework to enable representation learning without any supervised signal. In contrast, the introduced diffusion-based representation learning relies on a new formulation of the denoising score matching objective. Using the same approach, we propose to learn an infinite-dimensional latent code that achieves improvements of state-of-the-art models on semi-supervised image classification.
  arXiv  Detail & Related papers  (2021-05-29T09:26:02Z)
• Training Stacked Denoising Autoencoders for Representation Learning [0.0]
  We implement stacked autoencoders, a class of neural networks that are capable of learning powerful representations of high dimensional data. We describe gradient descent for unsupervised training of autoencoders, as well as a novel genetic algorithm based approach that makes use of gradient information.
  arXiv  Detail & Related papers  (2021-02-16T08:18:22Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• Unsupervised Audiovisual Synthesis via Exemplar Autoencoders [59.13989658692953]
  We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers. We use Exemplar Autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target speech exemplar.
  arXiv  Detail & Related papers  (2020-01-13T18:56:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.