A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms

• URL: http://arxiv.org/abs/2506.04419v1
• Date: Wed, 04 Jun 2025 19:58:51 GMT
• Title: A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms
• Abstract summary: We present a framework for auditing large language models (LLM)-based chatbots for dialect bias.<n>We measure the extent to which they produce quality-of-service harms, which occur when systems do not work equally well for different people.<n>Our framework has three key characteristics that make it useful in practice.
• Score: 0.6144680854063939
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Increasingly, individuals who engage in online activities are expected to interact with large language model (LLM)-based chatbots. Prior work has shown that LLMs can display dialect bias, which occurs when they produce harmful responses when prompted with text written in minoritized dialects. However, whether and how this bias propagates to systems built on top of LLMs, such as chatbots, is still unclear. We conduct a review of existing approaches for auditing LLMs for dialect bias and show that they cannot be straightforwardly adapted to audit LLM-based chatbots due to issues of substantive and ecological validity. To address this, we present a framework for auditing LLM-based chatbots for dialect bias by measuring the extent to which they produce quality-of-service harms, which occur when systems do not work equally well for different people. Our framework has three key characteristics that make it useful in practice. First, by leveraging dynamically generated instead of pre-existing text, our framework enables testing over any dialect, facilitates multi-turn conversations, and represents how users are likely to interact with chatbots in the real world. Second, by measuring quality-of-service harms, our framework aligns audit results with the real-world outcomes of chatbot use. Third, our framework requires only query access to an LLM-based chatbot, meaning that it can be leveraged equally effectively by internal auditors, external auditors, and even individual users in order to promote accountability. To demonstrate the efficacy of our framework, we conduct a case study audit of Amazon Rufus, a widely-used LLM-based chatbot in the customer service domain. Our results reveal that Rufus produces lower-quality responses to prompts written in minoritized English dialects, and that these quality-of-service harms are exacerbated by the presence of typos in prompts.

Related papers

• Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios [63.167237308464536]
  Speech translation (ST) is increasingly adopted in user applications, yet its evaluation focuses on decontextualized testbeds and holistic quality.<n>We introduce Ouvia, an evaluation framework for measuring user-perceived usability of speech translation outputs in real-world settings.
  arXiv  Detail & Related papers  (2026-06-04T13:52:21Z)
• Beyond Single-Policy: Evaluating Composed Organization-Specific Policy Alignment in LLM Chatbots [17.833004955025125]
  COPAL is an automated tool for evaluating composed-policy alignment in chatbots.<n>It generates queries that trigger composed-policy failures via empirically derived interaction patterns and explicit handling contracts.
  arXiv  Detail & Related papers  (2026-06-03T03:17:26Z)
• End-to-End Chatbot Evaluation with Adaptive Reasoning and Uncertainty Filtering [3.2434811678562685]
  Large language models (LLMs) combined with retrieval augmented generation have enabled the deployment of domain-specific chatbots, but these systems remain prone to generating unsupported or incorrect answers.<n>We propose an end-to-end automatic evaluator designed to substantially reduce human effort.<n>Our system generates Q&A pairs directly from the underlying knowledge base, uses LLMs to judge responses against reference answers, and applies confidence-based filtering to highlight uncertain cases.
  arXiv  Detail & Related papers  (2026-03-11T09:25:35Z)
• MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Chatbots and Dialogue Evaluators [8.672875654352689]
  This paper introduces MEDAL, an automated multi-agent framework for generating, evaluating, and curating dialogue evaluation benchmarks.<n>We generate user-chatbot multilingual dialogues conditioned on varied seed contexts.<n>A strong LLM is then used for a multidimensional analysis of the performance of the chatbots, uncovering noticeable cross-lingual performance differences.<n>A benchmark is then used to assess the ability of several reasoning and non-reasoning LLMs to act as evaluators of open-domain dialogues.
  arXiv  Detail & Related papers  (2025-05-28T18:45:42Z)
• One Language, Many Gaps: Evaluating Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks [68.33068005789116]
  We present the first study aimed at objectively assessing the fairness and robustness of Large Language Models (LLMs) in handling dialects in canonical reasoning tasks.<n>We hire AAVE speakers, including experts with computer science backgrounds, to rewrite seven popular benchmarks, such as HumanEval and GSM8K.<n>Our findings reveal that textbfalmost all of these widely used models show significant brittleness and unfairness to queries in AAVE.
  arXiv  Detail & Related papers  (2024-10-14T18:44:23Z)
• Prompt Engineering a Schizophrenia Chatbot: Utilizing a Multi-Agent Approach for Enhanced Compliance with Prompt Instructions [0.0699049312989311]
  Patients with schizophrenia often present with cognitive impairments that may hinder their ability to learn about their condition. While Large Language Models (LLMs) have the potential to make topical mental health information more accessible and engaging, their black-box nature raises concerns about ethics and safety.
  arXiv  Detail & Related papers  (2024-10-10T09:49:24Z)
• A Reliable Common-Sense Reasoning Socialbot Built Using LLMs and Goal-Directed ASP [3.17686396799427]
  We propose AutoCompanion, a socialbot that uses an LLM model to translate natural language into predicates. This paper presents the framework design and how an LLM is used to parse user messages and generate a response from the s(CASP) engine output.
  arXiv  Detail & Related papers  (2024-07-26T04:13:43Z)
• On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots [19.423566424346166]
  We study the use of Large Language Model (LLM)-based chatbots to power recommender systems. We observe that the chatbots respond poorly when they encounter under-specified requests. We conjecture that such miscalibrated response tendencies can be attributed to LLM fine-tuning using annotators.
  arXiv  Detail & Related papers  (2024-06-01T15:54:45Z)
• Hidden in Plain Sight: Exploring Chat History Tampering in Interactive Language Models [12.920884182101142]
  Large Language Models (LLMs) have become prevalent in real-world applications, exhibiting impressive text generation performance. To behave interactively, LLM-based chat systems must integrate prior chat history as context into their inputs, following a pre-defined structure. This paper introduces a systematic methodology to inject user-supplied history into LLM conversations without any prior knowledge of the target model.
  arXiv  Detail & Related papers  (2024-05-30T16:36:47Z)
• Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training [55.321010757641524]
  A primary concern regarding training large language models (LLMs) is whether they abuse copyrighted online text.<n>We propose an alternative textitinsert-and-detect methodology, advocating that web users and content platforms employ textbftextitunique identifiers for reliable and independent membership inference.
  arXiv  Detail & Related papers  (2024-03-23T06:36:32Z)
• SPML: A DSL for Defending Language Models Against Prompt Attacks [4.511923587827302]
  System Prompt Meta Language (SPML) is a domain-specific language for refining prompts and monitoring inputs to large language models (LLMs) SPML actively checks attack prompts, ensuring user inputs align with definitions to prevent malicious execution on the LLM backbone, optimizing costs. We introduce a groundbreaking benchmark with 1.8k system prompts and 20k user inputs, offering the inaugural language and benchmark for chatbots definition evaluation.
  arXiv  Detail & Related papers  (2024-02-19T00:53:48Z)
• CLOMO: Counterfactual Logical Modification with Large Language Models [109.60793869938534]
  We introduce a novel task, Counterfactual Logical Modification (CLOMO), and a high-quality human-annotated benchmark. In this task, LLMs must adeptly alter a given argumentative text to uphold a predetermined logical relationship. We propose an innovative evaluation metric, the Self-Evaluation Score (SES), to directly evaluate the natural language output of LLMs.
  arXiv  Detail & Related papers  (2023-11-29T08:29:54Z)
• BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues [72.65163468440434]
  This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting. We prompt large language models (LLMs) to generate a full multi-turn dialogue based on the ChatSEED, utterance by utterance. We find GPT-4 can generate human-style multi-turn dialogues with impressive quality, significantly outperforms its counterparts.
  arXiv  Detail & Related papers  (2023-10-20T16:53:51Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  We propose a novel rational evaluation approach that leverages pre-trained reward models as diagnostic tools. Longer conversations manifest the comprehensive grasp of language models in terms of their proficiency in understanding questions. Our results demonstrate that LLMs frequently exhibit vulnerability to word-level perturbations that are commonplace in daily language usage.
  arXiv  Detail & Related papers  (2023-09-20T09:23:46Z)
• DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection [56.513637720967566]
  Large language models (LLMs) can generate texts that pose risks of misuse, such as plagiarism, planting fake reviews on e-commerce platforms, or creating inflammatory false tweets. Existing high-quality detection methods usually require access to the interior of the model to extract the intrinsic characteristics. We propose to extract deep intrinsic characteristics of the black-box model generated texts.
  arXiv  Detail & Related papers  (2023-05-21T17:26:16Z)
• Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs [59.74002011562726]
  We propose a novel linguistic cue-based chain-of-thoughts (textitCue-CoT) to provide a more personalized and engaging response. We build a benchmark with in-depth dialogue questions, consisting of 6 datasets in both Chinese and English. Empirical results demonstrate our proposed textitCue-CoT method outperforms standard prompting methods in terms of both textithelpfulness and textitacceptability on all datasets.
  arXiv  Detail & Related papers  (2023-05-19T16:27:43Z)
• Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis: The case of apology [9.941695905504282]
  This study explores the possibility of using large language models (LLMs) to automate pragma-discursive corpus annotation.<n>We find that GPT-4 outperformed GPT-3.5, with accuracy approaching that of a human coder.
  arXiv  Detail & Related papers  (2023-05-15T04:10:13Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback [127.75419038610455]
  Large language models (LLMs) are able to generate human-like, fluent responses for many downstream tasks. This paper proposes a LLM-Augmenter system, which augments a black-box LLM with a set of plug-and-play modules.
  arXiv  Detail & Related papers  (2023-02-24T18:48:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.