Audio Dialogues: Dialogues dataset for audio and music understanding

• URL: http://arxiv.org/abs/2404.07616v1
• Date: Thu, 11 Apr 2024 10:08:34 GMT
• Title: Audio Dialogues: Dialogues dataset for audio and music understanding
• Authors: Arushi Goel, Zhifeng Kong, Rafael Valle, Bryan Catanzaro,
• Abstract summary: We introduce Audio Dialogues: a multi-turn dialogue dataset containing 163.8k samples for general audio sounds and music. In addition to dialogues, Audio Dialogues also has question-answer pairs to understand and compare multiple input audios together.
• Score: 29.550656226658962
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing datasets for audio understanding primarily focus on single-turn interactions (i.e. audio captioning, audio question answering) for describing audio in natural language, thus limiting understanding audio via interactive dialogue. To address this gap, we introduce Audio Dialogues: a multi-turn dialogue dataset containing 163.8k samples for general audio sounds and music. In addition to dialogues, Audio Dialogues also has question-answer pairs to understand and compare multiple input audios together. Audio Dialogues leverages a prompting-based approach and caption annotations from existing datasets to generate multi-turn dialogues using a Large Language Model (LLM). We evaluate existing audio-augmented large language models on our proposed dataset to demonstrate the complexity and applicability of Audio Dialogues. Our code for generating the dataset will be made publicly available. Detailed prompts and generated dialogues can be found on the demo website https://audiodialogues.github.io/.

Related papers

• Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models [58.43486430996411]
  Large Audio-Language Models (LALMs) have unclocked audio dialogue capabilities, where audio dialogues are a direct exchange of spoken language between LALMs and humans. Recent advances, such as GPT-4o, have enabled LALMs in back-and-forth audio dialogues with humans. We propose an Audio Dialogue Understanding Benchmark (ADU-Bench) to evaluate the performance of LALMs in the open-ended audio dialogue understanding.
  arXiv  Detail & Related papers  (2024-12-06T16:34:15Z)
• Music Discovery Dialogue Generation Using Human Intent Analysis and Large Language Models [10.022036983890091]
  We present a data generation framework for rich music discovery dialogue using a large language model (LLM) and user intents, system actions, and musical attributes. By applying this framework to the Million Song dataset, we create LP-MusicDialog, a Large Language Model based Pseudo Music Dialogue dataset. Our evaluation shows that the synthetic dataset is competitive with an existing, small human dialogue dataset.
  arXiv  Detail & Related papers  (2024-11-11T23:40:45Z)
• Identifying Speakers in Dialogue Transcripts: A Text-based Approach Using Pretrained Language Models [83.7506131809624]
  We introduce an approach to identifying speaker names in dialogue transcripts, a crucial task for enhancing content accessibility and searchability in digital media archives. We present a novel, large-scale dataset derived from the MediaSum corpus, encompassing transcripts from a wide range of media sources. We propose novel transformer-based models tailored for SpeakerID, leveraging contextual cues within dialogues to accurately attribute speaker names.
  arXiv  Detail & Related papers  (2024-07-16T18:03:58Z)
• Investigating the Effects of Large-Scale Pseudo-Stereo Data and Different Speech Foundation Model on Dialogue Generative Spoken Language Model [47.67067056593085]
  We develop a pipeline capable of transforming single-channel dialogue data into pseudo-stereo data. This expanded our training dataset from a mere 2,000 to an impressive 17,600 hours. The inclusion of this pseudo-stereo data has proven to be effective in improving the performance of spoken dialogue language models.
  arXiv  Detail & Related papers  (2024-07-02T03:22:41Z)
• Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation [55.043492250775294]
  We introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response. We also introduce MultiDialog, the first large-scale multimodal spoken dialogue corpus.
  arXiv  Detail & Related papers  (2024-06-12T04:48:36Z)
• WavJourney: Compositional Audio Creation with Large Language Models [38.39551216587242]
  We present WavJourney, a novel framework that leverages Large Language Models to connect various audio models for audio creation. WavJourney allows users to create storytelling audio content with diverse audio elements simply from textual descriptions. We show that WavJourney is capable of synthesizing realistic audio aligned with textually-described semantic, spatial and temporal conditions.
  arXiv  Detail & Related papers  (2023-07-26T17:54:04Z)
• DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI [92.29874802394167]
  DialogStudio is the largest and most diverse collection of dialogue datasets. Our collection encompasses data from open-domain dialogues, task-oriented dialogues, natural language understanding, conversational recommendation, dialogue summarization, and knowledge-grounded dialogues.
  arXiv  Detail & Related papers  (2023-07-19T17:57:53Z)
• VScript: Controllable Script Generation with Audio-Visual Presentation [56.17400243061659]
  VScript is a controllable pipeline that generates complete scripts including dialogues and scene descriptions. We adopt a hierarchical structure, which generates the plot, then the script and its audio-visual presentation. Experiment results show that our approach outperforms the baselines on both automatic and human evaluations.
  arXiv  Detail & Related papers  (2022-03-01T09:43:02Z)
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts [35.57757367869986]
  We release bf OpenViDial, a large-scale multi- module dialogue dataset. OpenViDial contains a total number of 1.1 million dialogue turns. We propose a family of encoder-decoder models leveraging both textual and visual contexts.
  arXiv  Detail & Related papers  (2020-12-30T03:02:50Z)
• Interview: A Large-Scale Open-Source Corpus of Media Dialog [11.28504775964698]
  We introduce 'Interview': a large-scale (105K conversations) media dialog dataset collected from news interview transcripts. Compared to existing large-scale proxies for conversational data, language models trained on our dataset exhibit better zero-shot out-of-domain performance. 'Interview' contains speaker role annotations for each turn, facilitating the development of engaging, responsive dialog systems.
  arXiv  Detail & Related papers  (2020-04-07T02:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.