MutaBot: A Mutation Testing Approach for Chatbots

• URL: http://arxiv.org/abs/2401.10372v1
• Date: Thu, 18 Jan 2024 20:38:27 GMT
• Title: MutaBot: A Mutation Testing Approach for Chatbots
• Authors: Michael Ferdinando Urrico, Diego Clerissi, Leonardo Mariani
• Abstract summary: MutaBot addresses mutations at multiple levels, including conversational flows, intents, and contexts. We assess the tool with three Dialogflow chatbots and test cases generated with Botium, revealing weaknesses in the test suites.
• Score: 3.811067614153878
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mutation testing is a technique aimed at assessing the effectiveness of test suites by seeding artificial faults into programs. Although available for many platforms and languages, no mutation testing tool is currently available for conversational chatbots, which represent an increasingly popular solution to design systems that can interact with users through a natural language interface. Note that since conversations must be explicitly engineered by the developers of conversational chatbots, these systems are exposed to specific types of faults not supported by existing mutation testing tools. In this paper, we present MutaBot, a mutation testing tool for conversational chatbots. MutaBot addresses mutations at multiple levels, including conversational flows, intents, and contexts. We designed the tool to potentially target multiple platforms, while we implemented initial support for Google Dialogflow chatbots. We assessed the tool with three Dialogflow chatbots and test cases generated with Botium, revealing weaknesses in the test suites.

Related papers

• Self-Directed Turing Test for Large Language Models [56.64615470513102]
  The Turing test examines whether AIs can exhibit human-like behaviour in natural language conversations. Traditional Turing tests adopt a rigid dialogue format where each participant sends only one message each time. This paper proposes the Self-Directed Turing Test, which extends the original test with a burst dialogue format.
  arXiv  Detail & Related papers  (2024-08-19T09:57:28Z)
• Automatically generating decision-support chatbots based on DMN models [0.0]
  We propose an approach for the automatic generation of fully functional, ready-to-use decisions-support chatbots based on a DNM decision model. With the aim of reducing chatbots development time and to allowing non-technical users the possibility of developing chatbots specific to their domain, all necessary phases were implemented in the Demabot tool.
  arXiv  Detail & Related papers  (2024-05-15T18:13:09Z)
• Measuring and Controlling Instruction (In)Stability in Language Model Dialogs [72.38330196290119]
  System-prompting is a tool for customizing language-model chatbots, enabling them to follow a specific instruction. We propose a benchmark to test the assumption, evaluating instruction stability via self-chats. We reveal a significant instruction drift within eight rounds of conversations. We propose a lightweight method called split-softmax, which compares favorably against two strong baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:10:29Z)
• ChatDev: Communicative Agents for Software Development [84.90400377131962]
  ChatDev is a chat-powered software development framework in which specialized agents are guided in what to communicate. These agents actively contribute to the design, coding, and testing phases through unified language-based communication.
  arXiv  Detail & Related papers  (2023-07-16T02:11:34Z)
• Understanding Multi-Turn Toxic Behaviors in Open-Domain Chatbots [8.763670548363443]
  A new attack, toxicbot, is developed to generate toxic responses in a multi-turn conversation. toxicbot can be used by both industry and researchers to develop methods for detecting and mitigating toxic responses in conversational dialogue.
  arXiv  Detail & Related papers  (2023-07-14T03:58:42Z)
• Developing Effective Educational Chatbots with ChatGPT prompts: Insights from Preliminary Tests in a Case Study on Social Media Literacy (with appendix) [43.55994393060723]
  Recent advances in language learning models with zero-shot learning capabilities, such as ChatGPT, suggest a new possibility for developing educational chatbots. We present a case study with a simple system that enables mixed-turn chatbots interactions. We examine ChatGPT's ability to pursue multiple interconnected learning objectives, adapt the educational activity to users' characteristics, such as culture, age, and level of education, and its ability to use diverse educational strategies and conversational styles.
  arXiv  Detail & Related papers  (2023-06-18T22:23:18Z)
• Implementing a Chatbot Solution for Learning Management System [0.0]
  One of the main problem that chatbots face today is to mimic human language. Extreme programming methodology was chosen to use integrate ChatterBot, Pyside2, web scraping and Tampermonkey into Blackboard. We showed the plausibility of integrating an AI bot in an educational setting.
  arXiv  Detail & Related papers  (2022-06-27T11:04:42Z)
• CheerBots: Chatbots toward Empathy and Emotionusing Reinforcement Learning [60.348822346249854]
  This study presents a framework whereby several empathetic chatbots are based on understanding users' implied feelings and replying empathetically for multiple dialogue turns. We call these chatbots CheerBots. CheerBots can be retrieval-based or generative-based and were finetuned by deep reinforcement learning. To respond in an empathetic way, we develop a simulating agent, a Conceptual Human Model, as aids for CheerBots in training with considerations on changes in user's emotional states in the future to arouse sympathy.
  arXiv  Detail & Related papers  (2021-10-08T07:44:47Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)
• If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills [4.395837214164745]
  It is challenging to build effective interview chatbots that can handle user free-text responses to open-ended questions. We are investigating the feasibility and effectiveness of using publicly available, practical AI technologies to build effective interview chatbots.
  arXiv  Detail & Related papers  (2020-02-05T16:52:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.