Story2MIDI: Emotionally Aligned Music Generation from Text

• URL: http://arxiv.org/abs/2512.02192v1
• Date: Mon, 01 Dec 2025 20:35:18 GMT
• Title: Story2MIDI: Emotionally Aligned Music Generation from Text
• Authors: Mohammad Shokri, Alexandra C. Salem, Gabriel Levine, Johanna Devaney, Sarah Ita Levitan,
• Abstract summary: We introduce Story2MIDI, a sequence-to-sequence Transformer-based model for generating emotion-aligned music from a given piece of text.<n>Our results indicate that our model effectively learns emotion-relevant features in music and incorporates them into its generation process.
• Score: 38.36870481571071
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce Story2MIDI, a sequence-to-sequence Transformer-based model for generating emotion-aligned music from a given piece of text. To develop this model, we construct the Story2MIDI dataset by merging existing datasets for sentiment analysis from text and emotion classification in music. The resulting dataset contains pairs of text blurbs and music pieces that evoke the same emotions in the reader or listener. Despite the small scale of our dataset and limited computational resources, our results indicate that our model effectively learns emotion-relevant features in music and incorporates them into its generation process, producing samples with diverse emotional responses. We evaluate the generated outputs using objective musical metrics and a human listening study, confirming the model's ability to capture intended emotional cues.

Related papers

• From Joy to Fear: A Benchmark of Emotion Estimation in Pop Song Lyrics [40.12543056558646]
  The emotional content of song lyrics plays a pivotal role in shaping listener experiences and influencing musical preferences.<n>This paper investigates the task of multi-label emotional attribution of song lyrics by predicting six emotional intensity scores corresponding to six fundamental emotions.
  arXiv  Detail & Related papers  (2025-09-06T06:28:28Z)
• Taming Transformer for Emotion-Controllable Talking Face Generation [61.835295250047196]
  We propose a novel method to tackle the emotion-controllable talking face generation task discretely.<n>Specifically, we employ two pre-training strategies to disentangle audio into independent components and quantize videos into combinations of visual tokens.<n>We conduct experiments on the MEAD dataset that controls the emotion of videos conditioned on multiple emotional audios.
  arXiv  Detail & Related papers  (2025-08-20T02:16:52Z)
• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation [75.86473375730392]
  SongGen is a fully open-source, single-stage auto-regressive transformer for controllable song generation.<n>It supports two output modes: mixed mode, which generates a mixture of vocals and accompaniment directly, and dual-track mode, which synthesizes them separately.<n>To foster community engagement and future research, we will release our model weights, training code, annotated data, and preprocessing pipeline.
  arXiv  Detail & Related papers  (2025-02-18T18:52:21Z)
• Are We There Yet? A Brief Survey of Music Emotion Prediction Datasets, Models and Outstanding Challenges [9.62904012066486]
  We provide a comprehensive overview of the available music-emotion datasets and discuss evaluation standards as well as competitions in the field.<n>We highlight the challenges that persist in accurately capturing emotion in music, including issues related to dataset quality, annotation consistency, and model generalization.<n>We argue that future advancements in music emotion recognition require standardized benchmarks, larger and more diverse datasets, and improved model interpretability.
  arXiv  Detail & Related papers  (2024-06-13T05:00:27Z)
• Emotion Manipulation Through Music -- A Deep Learning Interactive Visual Approach [0.0]
  We introduce a novel way to manipulate the emotional content of a song using AI tools. Our goal is to achieve the desired emotion while leaving the original melody as intact as possible. This research may contribute to on-demand custom music generation, the automated remixing of existing work, and music playlists tuned for emotional progression.
  arXiv  Detail & Related papers  (2024-06-12T20:12:29Z)
• MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models [57.47799823804519]
  We are inspired by how musicians compose music not just from a movie script, but also through visualizations. We propose MeLFusion, a model that can effectively use cues from a textual description and the corresponding image to synthesize music. Our exhaustive experimental evaluation suggests that adding visual information to the music synthesis pipeline significantly improves the quality of generated music.
  arXiv  Detail & Related papers  (2024-06-07T06:38:59Z)
• Exploring and Applying Audio-Based Sentiment Analysis in Music [0.0]
  The ability of a computational model to interpret musical emotions is largely unexplored. This study seeks to (1) predict the emotion of a musical clip over time and (2) determine the next emotion value after the music in a time series to ensure seamless transitions.
  arXiv  Detail & Related papers  (2024-02-22T22:34:06Z)
• Emotion4MIDI: a Lyrics-based Emotion-Labeled Symbolic Music Dataset [1.3607388598209322]
  We present a new large-scale emotion-labeled symbolic music dataset consisting of 12k MIDI songs. We first trained emotion classification models on the GoEmotions dataset, achieving state-of-the-art results with a model half the size of the baseline. Our dataset covers a wide range of fine-grained emotions, providing a valuable resource to explore the connection between music and emotions.
  arXiv  Detail & Related papers  (2023-07-27T11:24:47Z)
• Simple and Controllable Music Generation [94.61958781346176]
  MusicGen is a single Language Model (LM) that operates over several streams of compressed discrete music representation, i.e., tokens. Unlike prior work, MusicGen is comprised of a single-stage transformer LM together with efficient token interleaving patterns.
  arXiv  Detail & Related papers  (2023-06-08T15:31:05Z)
• Noise2Music: Text-conditioned Music Generation with Diffusion Models [73.74580231353684]
  We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models.
  arXiv  Detail & Related papers  (2023-02-08T07:27:27Z)
• Bridging Music and Text with Crowdsourced Music Comments: A Sequence-to-Sequence Framework for Thematic Music Comments Generation [18.2750732408488]
  We exploit the crowd-sourced music comments to construct a new dataset and propose a sequence-to-sequence model to generate text descriptions of music. To enhance the authenticity and thematicity of generated texts, we propose a discriminator and a novel topic evaluator.
  arXiv  Detail & Related papers  (2022-09-05T14:51:51Z)
• EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model [56.75775793011719]
  We introduce and publicly release a Mandarin emotion speech dataset including 9,724 samples with audio files and its emotion human-labeled annotation. Unlike those models which need additional reference audio as input, our model could predict emotion labels just from the input text and generate more expressive speech conditioned on the emotion embedding. In the experiment phase, we first validate the effectiveness of our dataset by an emotion classification task. Then we train our model on the proposed dataset and conduct a series of subjective evaluations.
  arXiv  Detail & Related papers  (2021-06-17T08:34:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.