Don't Stop the Multi-Party! On Generating Synthetic Multi-Party Conversations with Constraints

• URL: http://arxiv.org/abs/2502.13592v1
• Date: Wed, 19 Feb 2025 10:10:43 GMT
• Title: Don't Stop the Multi-Party! On Generating Synthetic Multi-Party Conversations with Constraints
• Authors: Nicolò Penzo, Marco Guerini, Bruno Lepri, Goran Glavaš, Sara Tonelli,
• Abstract summary: Multi-Party Conversations (MPCs) are widely studied across disciplines, with social media as a primary data source due to their accessibility. This work explores the feasibility of generating diverse MPCs with instruction-tuned Large Language Models.
• Score: 11.566214724241798
• License:
• Abstract: Multi-Party Conversations (MPCs) are widely studied across disciplines, with social media as a primary data source due to their accessibility. However, these datasets raise privacy concerns and often reflect platform-specific properties. For example, interactions between speakers may be limited due to rigid platform structures (e.g., threads, tree-like discussions), which yield overly simplistic interaction patterns (e.g., as a consequence of ``reply-to'' links). This work explores the feasibility of generating diverse MPCs with instruction-tuned Large Language Models (LLMs) by providing deterministic constraints such as dialogue structure and participants' stance. We investigate two complementary strategies of leveraging LLMs in this context: (i.) LLMs as MPC generators, where we task the LLM to generate a whole MPC at once and (ii.) LLMs as MPC parties, where the LLM generates one turn of the conversation at a time, provided the conversation history. We next introduce an analytical framework to evaluate compliance with the constraints, content quality, and interaction complexity for both strategies. Finally, we assess the quality of obtained MPCs via human annotation and LLM-as-a-judge evaluations. We find stark differences among LLMs, with only some being able to generate high-quality MPCs. We also find that turn-by-turn generation yields better conformance to constraints and higher linguistic variability than generating MPCs in one pass. Nonetheless, our structural and qualitative evaluation indicates that both generation strategies can yield high-quality MPCs.

Related papers

• When One LLM Drools, Multi-LLM Collaboration Rules [98.71562711695991]
  We argue for multi-LLM collaboration to better represent the extensive diversity of data, skills, and people. We organize existing multi-LLM collaboration methods into a hierarchy, based on the level of access and information exchange. We envision multi-LLM collaboration as an essential path toward compositional intelligence and collaborative AI development.
  arXiv  Detail & Related papers  (2025-02-06T21:13:44Z)
• Scoring with Large Language Models: A Study on Measuring Empathy of Responses in Dialogues [3.2162648244439684]
  We develop a framework for investigating how effective Large Language Models are at measuring and scoring empathy of responses in dialogues. Our strategy is to approximate the performance of state-of-the-art and fine-tuned LLMs with explicit and explainable features. Our results show that when only using embeddings, it is possible to achieve performance close to that of generic LLMs.
  arXiv  Detail & Related papers  (2024-12-28T20:37:57Z)
• Multi-Party Supervised Fine-tuning of Language Models for Multi-Party Dialogue Generation [11.340007143339657]
  Large Language Models (LLM) are usually fine-tuned to participate in dyadic or two-party dialogues. In this work, we design a multi-party fine-tuning framework (MuPaS) for LLMs on the multi-party dialogue datasets.
  arXiv  Detail & Related papers  (2024-12-06T09:33:47Z)
• NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews [65.35458530702442]
  We focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN. LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions.
  arXiv  Detail & Related papers  (2024-11-21T01:37:38Z)
• Robots in the Middle: Evaluating LLMs in Dispute Resolution [0.0]
  We investigate whether large language models (LLMs) are able to analyze dispute conversations, select suitable intervention types, and generate appropriate intervention messages. Our results demonstrate the potential of integrating AI in online dispute resolution (ODR) platforms.
  arXiv  Detail & Related papers  (2024-10-09T16:51:10Z)
• Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models [66.24055500785657]
  Traditional turn-based chat systems prevent users from verbally interacting with system while it is generating responses. To overcome these limitations, we adapt existing LLMs to listen users while generating output and provide users with instant feedback. We build a dataset consisting of alternating time slices of queries and responses as well as covering typical feedback types in instantaneous interactions.
  arXiv  Detail & Related papers  (2024-06-22T03:20:10Z)
• LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models [56.25156596019168]
  This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs) Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
  arXiv  Detail & Related papers  (2023-11-30T03:59:31Z)
• BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues [72.65163468440434]
  This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting. We prompt large language models (LLMs) to generate a full multi-turn dialogue based on the ChatSEED, utterance by utterance. We find GPT-4 can generate human-style multi-turn dialogues with impressive quality, significantly outperforms its counterparts.
  arXiv  Detail & Related papers  (2023-10-20T16:53:51Z)
• Let Models Speak Ciphers: Multiagent Debate through Embeddings [84.20336971784495]
  We introduce CIPHER (Communicative Inter-Model Protocol Through Embedding Representation) to address this issue. By deviating from natural language, CIPHER offers an advantage of encoding a broader spectrum of information without any modification to the model weights. This showcases the superiority and robustness of embeddings as an alternative "language" for communication among LLMs.
  arXiv  Detail & Related papers  (2023-10-10T03:06:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.