From Classification to Recommendation: Empirical Analysis of Audio Embedding Models Application for Content-Based Music Recommendation

• URL: http://arxiv.org/abs/2608.06928v2
• Date: Mon, 10 Aug 2026 05:16:34 GMT
• Title: From Classification to Recommendation: Empirical Analysis of Audio Embedding Models Application for Content-Based Music Recommendation
• Abstract summary: We evaluate six representative audio encoders across three types of music recommender systems.<n>We find that interaction-based sequential training substantially reduces performance differences among encoders.<n>We also find that increasing Semantic-ID capacity does not consistently improve generative recommender systems.
• Score: 21.60252248417606
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained audio representation models learned from large-scale corpora have achieved strong performance in audio classification and understanding. However, most existing models are optimized for objectives such as masked prediction, contrastive learning, or audio-text alignment, which do not necessarily produce representation spaces well-suited to recommender systems. Unlike classification, music recommender systems must capture item relationships shaped by subjective and behavior-dependent listener preferences. Although pretrained audio embeddings have been explored in conventional recommender systems, their effectiveness in the rapidly emerging paradigm of generative recommender systems remains underexplored. To address this gap, we systematically evaluate six representative audio encoders across three types of music recommender systems: content-based, sequential, and Semantic-ID-based generative recommender systems. We further investigate how residual-quantization design, including codebook width, quantization depth, and retained Semantic-ID prefixes, affects the preservation of recommendation-relevant information. Experiments on two music recommendation datasets show that audio-text-aligned and music-domain representations are generally more effective when pretrained embedding geometry is used directly, whereas interaction-based sequential training substantially reduces performance differences among encoders. We also find that increasing Semantic-ID capacity does not consistently improve generative recommender systems and may introduce substantial instability. These findings provide practical guidance for selecting audio encoders and designing audio-derived Semantic IDs for modern music recommender systems.

Related papers

• VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation [79.34043102155964]
  We propose VA-Judger, a chain-of-thought omni-reward model for joint video-audio generation.<n>We show VA-Judger outperforms metric baselines in predicting human preferences on both in-domain and out-of-domain evaluations.
  arXiv  Detail & Related papers  (2026-08-19T06:53:14Z)
• READ More than What You See: Reinforcement Learning for Accurate and Coherent Audio Description Generations [48.760748753984764]
  We present READ, the first reinforcement-learning framework for training-based AD generation.<n> READ formulates AD as sequence-level optimization with reference-matching, length, and format rewards.<n>Our results highlight RL as a promising paradigm for accurate and coherent AD generation.
  arXiv  Detail & Related papers  (2026-06-22T02:05:38Z)
• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• AudioMosaic: Contrastive Masked Audio Representation Learning [53.52371029884106]
  We introduce textbfAudioMosaic, a contrastive learning-based audio encoder for general audio understanding.<n>AudioMosaic constructs positive pairs by applying structured time-frequency masking to spectrogram patches.<n>Experiments show that AudioMosaic achieves state-of-the-art performance on several standard audio benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T00:56:51Z)
• Adopting State-of-the-Art Pretrained Audio Representations for Music Recommender Systems [0.0]
  Music Information Retrieval (MIR) research community has released various models pretrained on large amounts of music data.<n>Music FM, Music2Vec, MERT, EncodecMAE, Jukebox, MusiCNN, MULE, MuQ and MuQ-MuLan are investigated.<n>We find that pretrained audio representations exhibit significant performance disparity between traditional MIR tasks and both hot and cold music recommendations.
  arXiv  Detail & Related papers  (2026-04-25T00:09:58Z)
• Revisiting Content-Based Music Recommendation: Efficient Feature Aggregation from Large-Scale Music Models [54.4270504928356]
  Music Recommendation Systems (MRSs) are a cornerstone of modern streaming platforms.<n>We propose TASTE, a comprehensive dataset and benchmarking framework designed to highlight the role of multimodal information in music recommendation.<n>By leveraging recent large-scale self-supervised music encoders, we demonstrate the substantial value of the extracted audio representations across recommendation tasks.
  arXiv  Detail & Related papers  (2026-02-10T15:24:41Z)
• Representation-Regularized Convolutional Audio Transformer for Audio Understanding [53.092757178419355]
  bootstrapping representations from scratch is computationally expensive, often requiring extensive training to converge.<n>We propose the Convolutional Audio Transformer (CAT), a unified framework designed to address these challenges.
  arXiv  Detail & Related papers  (2026-01-29T12:16:19Z)
• Bridging the Gap Between Semantic and User Preference Spaces for Multi-modal Music Representation Learning [10.558648773612191]
  We propose a novel Hierarchical Two-stage Contrastive Learning (HTCL) method that models similarity from the semantic perspective to the user perspective hierarchically.<n>We devise a scalable audio encoder and leverage a pre-trained BERT model as the text encoder to learn audio-text semantics via large-scale contrastive pre-training.
  arXiv  Detail & Related papers  (2025-05-29T09:50:07Z)
• Comparative Analysis of Pretrained Audio Representations in Music Recommender Systems [0.0]
  Music Information Retrieval (MIR) has proposed various models pretrained on large amounts of music data. transfer learning showcases the proven effectiveness of pretrained backend models with a broad spectrum of downstream tasks. Music Recommender Systems tend to favour traditional end-to-end neural network learning over pretrained models.
  arXiv  Detail & Related papers  (2024-09-13T17:03:56Z)
• Impression-Aware Recommender Systems [53.48892326556546]
  We present a systematic literature review on recommender systems using impressions.<n>We define a theoretical framework to delimit recommender systems using impressions and a novel paradigm for personalized recommendations, called impression-aware recommender systems.
  arXiv  Detail & Related papers  (2023-08-15T16:16:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.