Multimodal Music Recommendation System using LLMs

• URL: http://arxiv.org/abs/2606.00125v1
• Date: Thu, 28 May 2026 15:34:12 GMT
• Title: Multimodal Music Recommendation System using LLMs
• Abstract summary: We propose a framework for session-based music recommendation that enriches the LastFM-1K dataset with three complementary signals.<n>We adopt the E4SRec framework by extending it with multimodal features and different item ID encoder backbones.<n>Our experiments show that integrating content-based features improves over ID-only baselines up to 95% in terms of Recall and 79% in terms of NDCG.
• Score: 53.1178230380919
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Music recommendation systems typically treat songs as opaque tokens, relying on collaborative interaction histories which overlooks semantic or acoustic content. Prior work has explored LLM-augmented, multimodal, and text-enhanced approaches to sequential recommendation, and while some methods partially combine semantic, acoustic, or engagement signals, none jointly model all three within a unified LLM-based sequential reasoning framework that grounds recommendations in actual song content. In this work, we propose a multimodal framework for session-based music recommendation that enriches the LastFM-1K dataset with three complementary signals: (1) audio and lyric embeddings extracted using pretrained music and text representation models, (2) LLM-generated semantic metadata using the MGPHot annotation schema, and (3) listening completion ratios. We adopt the E4SRec framework by extending it with multimodal features and different item ID encoder backbones, including SASRec, BERT4Rec, and GRU4Rec. We further extend the LLM backbone option with LLaMa-2-13B, Qwen2.5-7B-Instruct, and LLaMa-3-70B in both zero-shot and fine-tuned settings. Our experiments show that integrating content-based features improves over ID-only baselines up to 95% in terms of Recall and 79% in terms of NDCG. Moreover, our experiments show that naive multimodal fusion does not always yield additive improvements, highlighting challenges in cross-modal integration. We release a large-scale multimodal benchmark for music recommendation.

Related papers

• Revisiting Content-Based Music Recommendation: Efficient Feature Aggregation from Large-Scale Music Models [54.4270504928356]
  Music Recommendation Systems (MRSs) are a cornerstone of modern streaming platforms.<n>We propose TASTE, a comprehensive dataset and benchmarking framework designed to highlight the role of multimodal information in music recommendation.<n>By leveraging recent large-scale self-supervised music encoders, we demonstrate the substantial value of the extracted audio representations across recommendation tasks.
  arXiv  Detail & Related papers  (2026-02-10T15:24:41Z)
• ViLLA-MMBench: A Unified Benchmark Suite for LLM-Augmented Multimodal Movie Recommendation [14.62192876151853]
  ViLLA-MMBench is a benchmark for multimodal movie recommendation.<n>It aligns dense item embeddings from three modalities: audio (block-level, i-vector), visual (CNN, AVF), and text.<n>Missing or sparse metadata is automatically enriched using state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-08-06T08:39:07Z)
• LeVo: High-Quality Song Generation with Multi-Preference Alignment [47.965028296133426]
  We introduce LeVo, a language model based framework consisting of LeLM and Music Codec.<n>LeVo is capable of parallel modeling of two types of tokens: mixed tokens, which represent the combined audio of vocals and accompaniment.<n>It employs two decoder-only transformers and a modular extension training strategy to prevent interference between different token types.
  arXiv  Detail & Related papers  (2025-06-09T07:57:24Z)
• CLaMR: Contextualized Late-Interaction for Multimodal Content Retrieval [70.9990850395981]
  We introduce CLaMR, a multimodal, late-interaction retriever that jointly indexes 4 modalities: video frames, transcribed speech, on-screen text, and metadata.<n>CLaMR is trained to enhance dynamic modality selection via two key innovations.
  arXiv  Detail & Related papers  (2025-06-06T15:02:30Z)
• Molar: Multimodal LLMs with Collaborative Filtering Alignment for Enhanced Sequential Recommendation [4.518104756199573]
  Molar is a sequential recommendation framework that integrates multiple content modalities with ID information to capture collaborative signals effectively.<n>By seamlessly combining multimodal content with collaborative filtering insights, Molar captures both user interests and contextual semantics, leading to superior recommendation accuracy.
  arXiv  Detail & Related papers  (2024-12-24T05:23:13Z)
• LLM-based Bi-level Multi-interest Learning Framework for Sequential Recommendation [54.396000434574454]
  We propose a novel multi-interest SR framework combining implicit behavioral and explicit semantic perspectives.<n>It includes two modules: the Implicit Behavioral Interest Module and the Explicit Semantic Interest Module.<n>Experiments on four real-world datasets validate the framework's effectiveness and practicality.
  arXiv  Detail & Related papers  (2024-11-14T13:00:23Z)
• LARP: Language Audio Relational Pre-training for Cold-Start Playlist Continuation [49.89372182441713]
  We introduce LARP, a multi-modal cold-start playlist continuation model. Our framework uses increasing stages of task-specific abstraction: within-track (language-audio) contrastive loss, track-track contrastive loss, and track-playlist contrastive loss.
  arXiv  Detail & Related papers  (2024-06-20T14:02:15Z)
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models [66.11184017840688]
  We introduce C3LLM, a novel framework combining three tasks of video-to-audio, audio-to-text, and text-to-audio together. C3LLM adapts the Large Language Model (LLM) structure as a bridge for aligning different modalities. Our method combines the previous tasks of audio understanding, video-to-audio generation, and text-to-audio generation together into one unified model.
  arXiv  Detail & Related papers  (2024-05-25T09:10:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.