Towards Reliable Real-time Opera Tracking: Combining Alignment with Audio Event Detectors to Increase Robustness

• URL: http://arxiv.org/abs/2006.11033v1
• Date: Fri, 19 Jun 2020 09:31:07 GMT
• Title: Towards Reliable Real-time Opera Tracking: Combining Alignment with Audio Event Detectors to Increase Robustness
• Abstract summary: We first apply a state-of-the-art audio alignment method based on online Dynamic Time-Warping to full-length recordings of a Mozart opera. We analyze the tracker's most severe errors and identify three common sources of problems specific to the opera scenario. We propose a combination of a DTW-based music tracker with specialized audio event detectors.
• Score: 7.817685358710508
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in real-time music score following have made it possible for machines to automatically track highly complex polyphonic music, including full orchestra performances. In this paper, we attempt to take this to an even higher level, namely, live tracking of full operas. We first apply a state-of-the-art audio alignment method based on online Dynamic Time-Warping (OLTW) to full-length recordings of a Mozart opera and, analyzing the tracker's most severe errors, identify three common sources of problems specific to the opera scenario. To address these, we propose a combination of a DTW-based music tracker with specialized audio event detectors (for applause, silence/noise, and speech) that condition the DTW algorithm in a top-down fashion, and show, step by step, how these detectors add robustness to the score follower. However, there remain a number of open problems which we identify as targets for ongoing and future research.

Related papers

• FIGMA: Towards FIne-Grained Music retrievAl [65.98380295254817]
  Retrieving music using natural language descriptions has improved with contrastive audio-text models such as CLAP, but current systems remain limited to coarse semantic queries.<n>We show that this limitation stems from the contrastive learning objective itself: despite being trained on long captions, CLAP-based models effectively utilize only the first few tokens.<n>We propose FIGMA, a multi-view contrastive architecture that addresses this limitation by jointly optimizing global audio-text alignment and frame-level, token-wise alignment.
  arXiv  Detail & Related papers  (2026-06-04T18:05:39Z)
• AI-Generated Music Detection in Broadcast Monitoring [1.1937220268355655]
  We introduce AI-OpenBMAT, the first dataset tailored to broadcast-style AI-music detection.<n>It contains 3,294 one-minute audio excerpts that follow the duration patterns and loudness relations of real television audio.<n>We benchmark a CNN baseline and state-of-the-art SpectTTTra models to assess SNR and duration robustness, and evaluate on a full broadcast scenario.
  arXiv  Detail & Related papers  (2026-02-06T16:08:01Z)
• Sounding Highlights: Dual-Pathway Audio Encoders for Audio-Visual Video Highlight Detection [3.6453477876255502]
  We propose a novel framework, Dual-Pathway AudioSums for Video Highlight Detection (DAViHD)<n>DAViHD is composed of a semantic pathway for content understanding and a dynamic pathway that captures spectro-temporal dynamics.<n>We achieve new state-of-the-art performance on the large-scale MrHi benchmark.
  arXiv  Detail & Related papers  (2026-02-03T07:32:56Z)
• RUMAA: Repeat-Aware Unified Music Audio Analysis for Score-Performance Alignment, Transcription, and Mistake Detection [17.45655063331199]
  RUMAA is a transformer-based framework for music performance analysis.<n>It unifies score-to-performance alignment, score-informed transcription, and mistake detection in a near end-to-end manner.
  arXiv  Detail & Related papers  (2025-07-16T12:13:13Z)
• Learning Musical Representations for Music Performance Question Answering [10.912207282129753]
  multimodal learning methods are incapable of dealing with fundamental problems within the music performances. Our primary backbone is designed to incorporate multimodal interactions within the context of music data. Our experiments show state-of-the-art effects on the Music AVQA datasets.
  arXiv  Detail & Related papers  (2025-02-10T17:41:57Z)
• Toward a More Complete OMR Solution [49.74172035862698]
  Optical music recognition aims to convert music notation into digital formats. One approach to tackle OMR is through a multi-stage pipeline, where the system first detects visual music notation elements in the image. We introduce a music object detector based on YOLOv8, which improves detection performance. Second, we introduce a supervised training pipeline that completes the notation assembly stage based on detection output.
  arXiv  Detail & Related papers  (2024-08-31T01:09:12Z)
• BandControlNet: Parallel Transformers-based Steerable Popular Music Generation with Fine-Grained Spatiotemporal Features [19.284531698181116]
  BandControlNet is designed to tackle the multiple music sequences and generate high-quality music samples conditioned to the giventemporal control features. The proposed BandControlNet outperforms other conditional music generation models on most objective metrics in terms of fidelity and inference speed. The subjective evaluations show trained on short datasets can generate music with comparable quality to state-of-the-art models, while outperforming significantly using BandControlNet.
  arXiv  Detail & Related papers  (2024-07-15T06:33:25Z)
• LARP: Language Audio Relational Pre-training for Cold-Start Playlist Continuation [49.89372182441713]
  We introduce LARP, a multi-modal cold-start playlist continuation model. Our framework uses increasing stages of task-specific abstraction: within-track (language-audio) contrastive loss, track-track contrastive loss, and track-playlist contrastive loss.
  arXiv  Detail & Related papers  (2024-06-20T14:02:15Z)
• End-to-End Real-World Polyphonic Piano Audio-to-Score Transcription with Hierarchical Decoding [4.604877755214193]
  Existing end-to-end piano A2S systems have been trained and evaluated with only synthetic data. We propose a sequence-to-sequence (Seq2Seq) model with a hierarchical decoder that aligns with the hierarchical structure of musical scores. We propose a two-stage training scheme, which involves pre-training the model using an expressive performance rendering system on synthetic audio, followed by fine-tuning the model using recordings of human performance.
  arXiv  Detail & Related papers  (2024-05-22T10:52:04Z)
• TIM: A Time Interval Machine for Audio-Visual Action Recognition [64.24297230981168]
  We address the interplay between the two modalities in long videos by explicitly modelling the temporal extents of audio and visual events. We propose the Time Interval Machine (TIM) where a modality-specific time interval poses as a query to a transformer encoder. We test TIM on three long audio-visual video datasets: EPIC-KITCHENS, Perception Test, and AVE.
  arXiv  Detail & Related papers  (2024-04-08T14:30:42Z)
• EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving [64.58258341591929]
  Auditory Referring Multi-Object Tracking (AR-MOT) is a challenging problem in autonomous driving. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. We establish the first set of large-scale AR-MOT benchmarks.
  arXiv  Detail & Related papers  (2024-02-28T12:50:16Z)
• Towards Robust and Truly Large-Scale Audio-Sheet Music Retrieval [4.722882736419499]
  Cross-modal deep learning is used to learn joint embedding spaces that link the two distinct modalities - audio and sheet music images. While there has been steady improvement on this front over the past years, a number of open problems still prevent large-scale employment of this methodology. We identify a set of main challenges on the road towards robust and large-scale cross-modal music retrieval in real scenarios.
  arXiv  Detail & Related papers  (2023-09-21T15:11:16Z)
• Self-Supervised Contrastive Learning for Robust Audio-Sheet Music Retrieval Systems [3.997809845676912]
  We show that self-supervised contrastive learning can mitigate the scarcity of annotated data from real music content. We employ the snippet embeddings in the higher-level task of cross-modal piece identification. In this work, we observe that the retrieval quality improves from 30% up to 100% when real music data is present.
  arXiv  Detail & Related papers  (2023-09-21T14:54:48Z)
• GETMusic: Generating Any Music Tracks with a Unified Representation and Diffusion Framework [58.64512825534638]
  Symbolic music generation aims to create musical notes, which can help users compose music. We introduce a framework known as GETMusic, with GET'' standing for GEnerate music Tracks'' GETScore represents musical notes as tokens and organizes tokens in a 2D structure, with tracks stacked vertically and progressing horizontally over time. Our proposed representation, coupled with the non-autoregressive generative model, empowers GETMusic to generate music with any arbitrary source-target track combinations.
  arXiv  Detail & Related papers  (2023-05-18T09:53:23Z)
• An Empirical Evaluation of End-to-End Polyphonic Optical Music Recognition [24.377724078096144]
  Piano and orchestral scores frequently exhibit polyphonic passages, which add a second dimension to the task. We propose two novel formulations for end-to-end polyphonic OMR. We observe a new state-of-the-art performance with our multi-sequence detection decoder, RNNDecoder.
  arXiv  Detail & Related papers  (2021-08-03T22:04:40Z)
• Distractor-Aware Fast Tracking via Dynamic Convolutions and MOT Philosophy [63.91005999481061]
  A practical long-term tracker typically contains three key properties, i.e. an efficient model design, an effective global re-detection strategy and a robust distractor awareness mechanism. We propose a two-task tracking frame work (named DMTrack) to achieve distractor-aware fast tracking via Dynamic convolutions (d-convs) and Multiple object tracking (MOT) philosophy. Our tracker achieves state-of-the-art performance on the LaSOT, OxUvA, TLP, VOT2018LT and VOT 2019LT benchmarks and runs in real-time (3x faster
  arXiv  Detail & Related papers  (2021-04-25T00:59:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.