A Model-based Testing Technique for Amazon Lex Task-based Chatbots

• URL: http://arxiv.org/abs/2607.01094v1
• Date: Wed, 01 Jul 2026 15:48:43 GMT
• Title: A Model-based Testing Technique for Amazon Lex Task-based Chatbots
• Abstract summary: LexTester is an automated model-based testing technique for Amazon Lex chatbots.<n>It generates a Dialog Graph of all possible interactions, from which an executable test suite is generated.
• Score: 46.12795846459242
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Task-based chatbots are nowadays widely adopted software systems, usually integrated into real-world applications and communication channels, designed to assist users in completing tasks through conversational interfaces. Like any other software, even chatbots are prone to bugs. Despite their increasing pervasiveness in everyday activities, existing techniques for assessing their quality still exhibit several limitations, such as the simplicity of generated test scenarios and oracle weaknesses. In this paper, we present LexTester, an automated model-based testing technique for Amazon Lex chatbots. The technique explores the conversational space of the chatbot under test to generate a Dialog Graph of all possible interactions, from which an executable test suite is generated according to different coverage strategies. LexTester was evaluated against the state-of-the-practice chatbot testing tool Botium on five Amazon Lex chatbots, consistently outperforming it in all subjects, generating more tests with nearly double complexity, achieving overall 83-95% coverage of conversational elements, and improving fault detection effectiveness by up to four times at comparable time costs.

Related papers

• WebTestBench: Evaluating Computer-Use Agents towards End-to-End Automated Web Testing [57.7131457251794]
  We introduce WebTestBench, a benchmark for evaluating end-to-end automated web testing.<n>We decompose the testing process into two cascaded sub-tasks, checklist generation and defect detection, and propose WebTester.<n>Our findings expose a substantial gap between current computer-use agent capabilities and industrial-grade deployment demands.
  arXiv  Detail & Related papers  (2026-03-26T09:27:29Z)
• Automated Testing of Task-based Chatbots: How Far Are We? [5.64612424709862]
  Task-based chatbots are software, typically embedded in real-world applications, that assist users in completing tasks through a conversational interface.<n>In this paper, we evaluate the effectiveness of state-of-the-art testing techniques on a curated selection of task-based chatbots from GitHub.
  arXiv  Detail & Related papers  (2026-02-13T16:32:50Z)
• Assessing Task-based Chatbots: Snapshot and Curated Datasets for Dialogflow [5.64612424709862]
  This paper presents TOFU-D, a snapshot of 1,788 Dialogflow chatbots from GitHub, and COD, a curated subset of TOFU-D including 185 validated chatbots.<n>A preliminary assessment using the Botium testing framework and the Bandit static analyzer revealed gaps in test coverage and frequent security vulnerabilities in several chatbots.
  arXiv  Detail & Related papers  (2026-01-27T16:49:56Z)
• Towards Multi-Platform Mutation Testing of Task-based Chatbots [5.64612424709862]
  We present our extension of MUTABOT to multiple platforms (Dialogflow and Rasa)<n>MUTABOT is a mutation testing approach for injecting faults in conversations.<n>We show how mutation testing can be used to reveal weaknesses in test suites generated by the Botium state-of-the-art test generator.
  arXiv  Detail & Related papers  (2025-09-01T11:36:06Z)
• Test Case Generation for Dialogflow Task-Based Chatbots [3.488620810035772]
  Test Generator (CTG) is an automated testing technique designed for task-based chatbots.<n>We conducted an experiment comparing CTG with state-of-the-art BOTIUM and CHARM tools.<n>CTG outperformed the competitors in terms of robustness and effectiveness.
  arXiv  Detail & Related papers  (2025-03-07T16:39:27Z)
• X-TURING: Towards an Enhanced and Efficient Turing Test for Long-Term Dialogue Agents [56.64615470513102]
  The Turing test examines whether AIs exhibit human-like behaviour in natural language conversations.<n>Traditional setting limits each participant to one message at a time and requires constant human participation.<n>This paper proposes textbftextscX-Turing, which enhances the original test with a textitburst dialogue pattern.
  arXiv  Detail & Related papers  (2024-08-19T09:57:28Z)
• MutaBot: A Mutation Testing Approach for Chatbots [3.811067614153878]
  MutaBot addresses mutations at multiple levels, including conversational flows, intents, and contexts. We assess the tool with three Dialogflow chatbots and test cases generated with Botium, revealing weaknesses in the test suites.
  arXiv  Detail & Related papers  (2024-01-18T20:38:27Z)
• Train Offline, Test Online: A Real Robot Learning Benchmark [113.19664479709587]
  Train Offline, Test Online (TOTO) provides remote users with access to shared robotic hardware for evaluating methods on common tasks. We present initial results on TOTO comparing five pretrained visual representations and four offline policy learning baselines, remotely contributed by five institutions. The real promise of TOTO, however, lies in the future: we release the benchmark for additional submissions from any user, enabling easy, direct comparison to several methods without the need to obtain hardware or collect data.
  arXiv  Detail & Related papers  (2023-06-01T17:42:08Z)
• Towards Robust Online Dialogue Response Generation [62.99904593650087]
  We argue that this can be caused by a discrepancy between training and real-world testing. We propose a hierarchical sampling-based method consisting of both utterance-level sampling and semi-utterance-level sampling.
  arXiv  Detail & Related papers  (2022-03-07T06:51:41Z)
• Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention [55.77218465471519]
  This paper proposes an innovative framework to train chatbots to possess human-like intentions. Our framework included a guiding robot and an interlocutor model that plays the role of humans. We examined our framework using three experimental setups and evaluate the guiding robot with four different metrics to demonstrated flexibility and performance advantages.
  arXiv  Detail & Related papers  (2021-03-30T15:24:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.