SocialDial: A Benchmark for Socially-Aware Dialogue Systems

• URL: http://arxiv.org/abs/2304.12026v1
• Date: Mon, 24 Apr 2023 11:55:22 GMT
• Title: SocialDial: A Benchmark for Socially-Aware Dialogue Systems
• Authors: Haolan Zhan and Zhuang Li and Yufei Wang and Linhao Luo and Tao Feng and Xiaoxi Kang and Yuncheng Hua and Lizhen Qu and Lay-Ki Soon and Suraj Sharma and Ingrid Zukerman and Zhaleh Semnani-Azad and Gholamreza Haffari
• Abstract summary: We present the first socially-aware dialogue corpus - SocialDial, based on Chinese social culture. SocialDial consists of two parts: 1,563 multi-turn dialogues between two human speakers with fine-grained labels, and 4,870 synthetic conversations generated by ChatGPT. The human corpus covers five categories of social norms, which have 14 sub-categories in total.
• Score: 45.3266270265532
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dialogue systems have been widely applied in many scenarios and are now more powerful and ubiquitous than ever before. With large neural models and massive available data, current dialogue systems have access to more knowledge than any people in their life. However, current dialogue systems still do not perform at a human level. One major gap between conversational agents and humans lies in their abilities to be aware of social norms. The development of socially-aware dialogue systems is impeded due to the lack of resources. In this paper, we present the first socially-aware dialogue corpus - SocialDial, based on Chinese social culture. SocialDial consists of two parts: 1,563 multi-turn dialogues between two human speakers with fine-grained labels, and 4,870 synthetic conversations generated by ChatGPT. The human corpus covers five categories of social norms, which have 14 sub-categories in total. Specifically, it contains social factor annotations including social relation, context, social distance, and social norms. However, collecting sufficient socially-aware dialogues is costly. Thus, we harness the power of ChatGPT and devise an ontology-based synthetic data generation framework. This framework is able to generate synthetic data at scale. To ensure the quality of synthetic dialogues, we design several mechanisms for quality control during data collection. Finally, we evaluate our dataset using several pre-trained models, such as BERT and RoBERTa. Comprehensive empirical results based on state-of-the-art neural models demonstrate that modeling of social norms for dialogue systems is a promising research direction. To the best of our knowledge, SocialDial is the first socially-aware dialogue dataset that covers multiple social factors and has fine-grained labels.

Related papers

• Social Orientation: A New Feature for Dialogue Analysis [15.192659799728181]
  We introduce a new data set of dialogue utterances machine-labeled with social orientation tags. We show that social orientation tags improve task performance, especially in low-resource settings. We also demonstrate how social orientation tags help explain the outcomes of social interactions when used in neural models.
  arXiv  Detail & Related papers  (2024-02-26T01:55:45Z)
• SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents [107.4138224020773]
  We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and humans. In our environment, agents role-play and interact under a wide variety of scenarios; they coordinate, collaborate, exchange, and compete with each other to achieve complex social goals. We find that GPT-4 achieves a significantly lower goal completion rate than humans and struggles to exhibit social commonsense reasoning and strategic communication skills.
  arXiv  Detail & Related papers  (2023-10-18T02:27:01Z)
• PLACES: Prompting Language Models for Social Conversation Synthesis [103.94325597273316]
  We use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations.
  arXiv  Detail & Related papers  (2023-02-07T05:48:16Z)
• SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization [129.1927527781751]
  We present SODA, the first publicly available, million-scale high-quality social dialogue dataset. By contextualizing social commonsense knowledge from a knowledge graph, we are able to distill an exceptionally broad spectrum of social interactions. Human evaluation shows that conversations in SODA are more consistent, specific, and (surprisingly) natural than those in prior human-authored datasets.
  arXiv  Detail & Related papers  (2022-12-20T17:38:47Z)
• ProsocialDialog: A Prosocial Backbone for Conversational Agents [104.92776607564583]
  We introduce ProsocialDialog, the first large-scale dialogue dataset to teach conversational agents to respond to problematic content following social norms. Created via a human-AI collaborative framework, ProsocialDialog consists of 58K dialogues, with 331K utterances, 160K RoTs, and 497K dialogue safety labels. With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost.
  arXiv  Detail & Related papers  (2022-05-25T11:48:47Z)
• A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots [0.0]
  We aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. We present a broad overview of methods developed to build dialogue systems over the years.
  arXiv  Detail & Related papers  (2021-11-02T08:07:55Z)
• SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents [23.719833581321033]
  Building embodied autonomous agents capable of participating in social interactions with humans is one of the main challenges in AI. We argue that aiming towards human-level AI requires a broader set of key social skills. We present SocialAI, a benchmark to assess the acquisition of social skills of DRL agents.
  arXiv  Detail & Related papers  (2021-07-02T10:39:18Z)
• Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents [60.27066549589362]
  Social language used by human agents is associated with greater users' responsiveness and task completion. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.
  arXiv  Detail & Related papers  (2020-12-29T08:22:48Z)
• Building A User-Centric and Content-Driven Socialbot [2.072266782237039]
  We develop a system architecture that is capable of accommodating dialog strategies that we designed for socialbot conversations. The architecture consists of a multi-dimensional language understanding module for analyzing user utterances. We construct a new knowledge base to power the socialbot by collecting social chat content from a variety of sources.
  arXiv  Detail & Related papers  (2020-05-06T07:11:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.