Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts

• URL: http://arxiv.org/abs/2405.01121v3
• Date: Tue, 15 Oct 2024 08:48:03 GMT
• Title: Efficient Data Generation for Source-grounded Information-seeking Dialogs: A Use Case for Meeting Transcripts
• Authors: Lotem Golany, Filippo Galgani, Maya Mamo, Nimrod Parasol, Omer Vandsburger, Nadav Bar, Ido Dagan,
• Abstract summary: We investigate the feasibility and effectiveness of Large Language Models (LLMs)-based data generation in source-grounded information-seeking dialogs. We create MISeD -- Meeting Information Seeking Dialogs dataset -- a dataset of information-seeking dialogs focused on meeting transcripts. Finetuning on MISeD gives comparable response generation quality to finetuning on fully manual data, while improving attribution quality and reducing time and effort.
• Score: 10.829227084902428
• License:
• Abstract: Automating data generation with Large Language Models (LLMs) has become increasingly popular. In this work, we investigate the feasibility and effectiveness of LLM-based data generation in the challenging setting of source-grounded information-seeking dialogs, with response attribution, over long documents. Our source texts consist of long and noisy meeting transcripts, adding to the task complexity. Since automating attribution remains difficult, we propose a semi-automatic approach: dialog queries and responses are generated with LLMs, followed by human verification and identification of attribution spans. Using this approach, we created MISeD -- Meeting Information Seeking Dialogs dataset -- a dataset of information-seeking dialogs focused on meeting transcripts. Models finetuned with MISeD demonstrate superior performance compared to off-the-shelf models, even those of larger size. Finetuning on MISeD gives comparable response generation quality to finetuning on fully manual data, while improving attribution quality and reducing time and effort.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation [12.93942316816741]
  GPT-4 is used to simulate the user and agent interaction, generating thousands of annotated dialogues with DST labels. A two-stage fine-tuning on LLaMA 2 is performed on the generated data and the real data for the DST prediction. Our approach is also capable of adapting to the dynamic demands in real-world scenarios, generating dialogues in new domains swiftly.
  arXiv  Detail & Related papers  (2024-05-17T07:00:05Z)
• Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models [16.94819621353007]
  SynTOD is a new synthetic data generation approach for developing end-to-end Task-Oriented Dialogue (TOD) systems. It generates diverse, structured conversations through random walks and response simulation using large language models. In our experiments, using graph-guided response simulations leads to significant improvements in intent classification, slot filling and response relevance.
  arXiv  Detail & Related papers  (2024-04-23T06:23:34Z)
• Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generation [6.685921135304385]
  We propose textbfSummary-based textbfDialogue textbfAugmentation with LLM. Our approach enhances the controllability of LLM by using dialogue summaries as a planning tool. Based on summaries, SDA can generate high-quality and diverse dialogue data even with a small seed dataset.
  arXiv  Detail & Related papers  (2024-03-30T13:28:51Z)
• LUCID: LLM-Generated Utterances for Complex and Interesting Dialogues [38.6183579217801]
  Virtual assistants are poised to take a leap forward in terms of their dialogue capabilities. Yet a major bottleneck to achieving genuinely transformative task-oriented dialogue capabilities remains the scarcity of high quality data. We use LUCID to generate a seed dataset of 4,277 conversations across 100 intents to demonstrate its capabilities.
  arXiv  Detail & Related papers  (2024-03-01T11:33:53Z)
• Evaluating Very Long-Term Conversational Memory of LLM Agents [95.84027826745609]
  We introduce a machine-human pipeline to generate high-quality, very long-term dialogues. We equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency.
  arXiv  Detail & Related papers  (2024-02-27T18:42:31Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)
• Learning towards Selective Data Augmentation for Dialogue Generation [52.540330534137794]
  We argue that not all cases are beneficial for augmentation task, and the cases suitable for augmentation should obey the following two attributes. We propose a Selective Data Augmentation framework (SDA) for the response generation task.
  arXiv  Detail & Related papers  (2023-03-17T01:26:39Z)
• Weakly Supervised Data Augmentation Through Prompting for Dialogue Understanding [103.94325597273316]
  We present a novel approach that iterates on augmentation quality by applying weakly-supervised filters. We evaluate our methods on the emotion and act classification tasks in DailyDialog and the intent classification task in Facebook Multilingual Task-Oriented Dialogue. For DailyDialog specifically, using 10% of the ground truth data we outperform the current state-of-the-art model which uses 100% of the data.
  arXiv  Detail & Related papers  (2022-10-25T17:01:30Z)
• Paraphrase Augmented Task-Oriented Dialog Generation [68.1790912977053]
  We propose a paraphrase augmented response generation (PARG) framework that jointly trains a paraphrase model and a response generation model. We also design a method to automatically construct paraphrase training data set based on dialog state and dialog act labels.
  arXiv  Detail & Related papers  (2020-04-16T05:12:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.