A Human-Computer Duet System for Music Performance

• URL: http://arxiv.org/abs/2009.07816v1
• Date: Wed, 16 Sep 2020 17:19:23 GMT
• Title: A Human-Computer Duet System for Music Performance
• Authors: Yuen-Jen Lin, Hsuan-Kai Kao, Yih-Chih Tseng, Ming Tsai, Li Su
• Abstract summary: We create a virtual violinist who can collaborate with a human pianist to perform chamber music automatically without any intervention. The system incorporates the techniques from various fields, including real-time music tracking, pose estimation, and body movement generation. The proposed system has been validated in public concerts.
• Score: 7.777761975348974
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Virtual musicians have become a remarkable phenomenon in the contemporary multimedia arts. However, most of the virtual musicians nowadays have not been endowed with abilities to create their own behaviors, or to perform music with human musicians. In this paper, we firstly create a virtual violinist, who can collaborate with a human pianist to perform chamber music automatically without any intervention. The system incorporates the techniques from various fields, including real-time music tracking, pose estimation, and body movement generation. In our system, the virtual musician's behavior is generated based on the given music audio alone, and such a system results in a low-cost, efficient and scalable way to produce human and virtual musicians' co-performance. The proposed system has been validated in public concerts. Objective quality assessment approaches and possible ways to systematically improve the system are also discussed.

Related papers

• Musical Agent Systems: MACAT and MACataRT [6.349140286855134]
  We introduce MACAT and MACataRT, two distinct musical agent systems crafted to enhance interactive music-making between human musicians and AI. MaCAT is optimized for agent-led performance, employing real-time synthesis and self-listening to shape its output autonomously. MacataRT provides a flexible environment for collaborative improvisation through audio mosaicing and sequence-based learning.
  arXiv  Detail & Related papers  (2025-01-19T22:04:09Z)
• Tuning Music Education: AI-Powered Personalization in Learning Music [0.2046223849354785]
  We present two case studies using such advances in music technology to address challenges in music education. In our first case study we showcase an application that uses Automatic Chord Recognition to generate personalized exercises from audio tracks. In the second case study we prototype adaptive piano method books that use Automatic Music Transcription to generate exercises at different skill levels.
  arXiv  Detail & Related papers  (2024-12-18T05:25:42Z)
• Music-triggered fashion design: from songs to the metaverse [32.73124984242397]
  We analyze how virtual realities can help to broaden the opportunities for musicians to bridge with their audiences. We present our first steps towards re-defining musical experiences in the metaverse.
  arXiv  Detail & Related papers  (2024-10-07T11:09:45Z)
• A Survey of Foundation Models for Music Understanding [60.83532699497597]
  This work is one of the early reviews of the intersection of AI techniques and music understanding. We investigated, analyzed, and tested recent large-scale music foundation models in respect of their music comprehension abilities.
  arXiv  Detail & Related papers  (2024-09-15T03:34:14Z)
• AffectMachine-Classical: A novel system for generating affective classical music [0.0]
  AffectMachine-Classical is capable of generating affective Classic music in real-time. A listener study was conducted to validate the ability of the system to reliably convey target emotions to listeners. Future work will embed AffectMachine-Classical into biofeedback systems, to leverage the efficacy of the affective music for emotional well-being in listeners.
  arXiv  Detail & Related papers  (2023-04-11T01:06:26Z)
• Quantized GAN for Complex Music Generation from Dance Videos [48.196705493763986]
  We present Dance2Music-GAN (D2M-GAN), a novel adversarial multi-modal framework that generates musical samples conditioned on dance videos. Our proposed framework takes dance video frames and human body motion as input, and learns to generate music samples that plausibly accompany the corresponding input.
  arXiv  Detail & Related papers  (2022-04-01T17:53:39Z)
• Flat latent manifolds for music improvisation between human and machine [9.571383193449648]
  We consider a music-generating algorithm as a counterpart to a human musician, in a setting where reciprocal improvisation is to lead to new experiences. In the learned model, we generate novel musical sequences by quantification in latent space. We provide empirical evidence for our method via a set of experiments on music and we deploy our model for an interactive jam session with a professional drummer.
  arXiv  Detail & Related papers  (2022-02-23T09:00:17Z)
• Music-to-Dance Generation with Optimal Transport [48.92483627635586]
  We propose a Music-to-Dance with Optimal Transport Network (MDOT-Net) for learning to generate 3D dance choreographs from music. We introduce an optimal transport distance for evaluating the authenticity of the generated dance distribution and a Gromov-Wasserstein distance to measure the correspondence between the dance distribution and the input music.
  arXiv  Detail & Related papers  (2021-12-03T09:37:26Z)
• Learning to Generate Diverse Dance Motions with Transformer [67.43270523386185]
  We introduce a complete system for dance motion synthesis. A massive dance motion data set is created from YouTube videos. A novel two-stream motion transformer generative model can generate motion sequences with high flexibility.
  arXiv  Detail & Related papers  (2020-08-18T22:29:40Z)
• Foley Music: Learning to Generate Music from Videos [115.41099127291216]
  Foley Music is a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements.
  arXiv  Detail & Related papers  (2020-07-21T17:59:06Z)
• Music Gesture for Visual Sound Separation [121.36275456396075]
  "Music Gesture" is a keypoint-based structured representation to explicitly model the body and finger movements of musicians when they perform music. We first adopt a context-aware graph network to integrate visual semantic context with body dynamics, and then apply an audio-visual fusion model to associate body movements with the corresponding audio signals.
  arXiv  Detail & Related papers  (2020-04-20T17:53:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.