Towards a Human-like Open-Domain Chatbot

• URL: http://arxiv.org/abs/2001.09977v3
• Date: Thu, 27 Feb 2020 07:36:47 GMT
• Title: Towards a Human-like Open-Domain Chatbot
• Abstract summary: This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We propose a human evaluation metric called Sensibleness and Specificity Average (SSA) which captures key elements of a human-like multi-turn conversation.
• Score: 56.94906081225566
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.

Related papers

• Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue [1.1305136905804842]
  We present Inverse Turing Bench, a benchmark that evaluates models on their ability to differentiate humans and AI in multi-turn text.<n>The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is between a human and an AI.<n>We evaluated a preliminary set of models against this benchmark, and found that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest accuracy.
  arXiv  Detail & Related papers  (2026-06-20T02:47:56Z)
• AudioInsight: Detecting Social Contexts Relevant to Social Anxiety from Speech [10.648118103594586]
  This study presents a novel approach that harnesses ambient audio segments to detect social threat contexts. We focus on two key dimensions: number of interaction partners (dyadic vs. group) and degree of evaluative threat. Under sample-wide 5-fold Cross Validation (CV), our model distinguished dyadic from group interactions with 90% accuracy and detected evaluative threat at 83%.
  arXiv  Detail & Related papers  (2024-07-19T17:01:12Z)
• From Words and Exercises to Wellness: Farsi Chatbot for Self-Attachment Technique [1.7592522344393486]
  We develop a voice-capable bot in Farsi to guide users through Self-Attachment (SAT) We collect a dataset of over 6,000 utterances and develop a novel sentiment-analysis module that classifies user sentiment into 12 classes, with accuracy above 92%. Our platform was engaging to most users (75%), 72% felt better after the interactions, and 74% were satisfied with the SAT Teacher's performance.
  arXiv  Detail & Related papers  (2023-10-13T19:09:31Z)
• Unsupervised Sentiment Analysis of Plastic Surgery Social Media Posts [91.3755431537592]
  The massive collection of user posts across social media platforms is primarily untapped for artificial intelligence (AI) use cases. Natural language processing (NLP) is a subfield of AI that leverages bodies of documents, known as corpora, to train computers in human-like language understanding. This study demonstrates that the applied results of unsupervised analysis allow a computer to predict either negative, positive, or neutral user sentiment towards plastic surgery.
  arXiv  Detail & Related papers  (2023-07-05T20:16:20Z)
• Evaluating Human-Language Model Interaction [79.33022878034627]
  We develop a new framework, Human-AI Language-based Interaction Evaluation (HALIE), that defines the components of interactive systems. We design five tasks to cover different forms of interaction: social dialogue, question answering, crossword puzzles, summarization, and metaphor generation. We find that better non-interactive performance does not always translate to better human-LM interaction.
  arXiv  Detail & Related papers  (2022-12-19T18:59:45Z)
• Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees [63.62448343531963]
  We propose a combination of the existing paradigms, sampling responses to be scored by humans intelligently. We observe significant gains in accuracy (19.80% increase on average) and quadratic weighted kappa (QWK) (25.60% on average) with a relatively small human budget.
  arXiv  Detail & Related papers  (2021-11-17T05:00:51Z)
• Training Conversational Agents with Generative Conversational Networks [74.9941330874663]
  We use Generative Conversational Networks to automatically generate data and train social conversational agents. We evaluate our approach on TopicalChat with automatic metrics and human evaluators, showing that with 10% of seed data it performs close to the baseline that uses 100% of the data.
  arXiv  Detail & Related papers  (2021-10-15T21:46:39Z)
• Low-Resource Adaptation of Open-Domain Generative Chatbots [0.0]
  We show that low parameter models can retain their general knowledge conversational abilities while improving in a specific domain. We propose a generic framework that accounts for variety in question types, tracks reference throughout multi-turn conversations, and removes inconsistent and potentially toxic responses. Our framework seamlessly transitions between chatting and performing transactional tasks, which will ultimately make interactions with digital assistants more human-like.
  arXiv  Detail & Related papers  (2021-08-13T17:40:30Z)
• UniCon: Unified Context Network for Robust Active Speaker Detection [111.90529347692723]
  We introduce a new efficient framework, the Unified Context Network (UniCon), for robust active speaker detection (ASD) Our solution is a novel, unified framework that focuses on jointly modeling multiple types of contextual information. A thorough ablation study is performed on several challenging ASD benchmarks under different settings.
  arXiv  Detail & Related papers  (2021-08-05T13:25:44Z)
• A Hybrid Solution to Learn Turn-Taking in Multi-Party Service-based Chat Groups [2.943984871413744]
  In a text-based chat group, the only information available is the sender, the content of the text and the dialogue history. We present our study on how these information can be used on the prediction task through a corpus and architecture.
  arXiv  Detail & Related papers  (2020-01-14T22:37:21Z)
• Hyperparameters optimization for Deep Learning based emotion prediction for Human Robot Interaction [0.2549905572365809]
  We have proposed an Inception module based Convolutional Neural Network Architecture. The model is implemented in a humanoid robot, NAO in real time and robustness of the model is evaluated.
  arXiv  Detail & Related papers  (2020-01-12T05:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.