Bita: A Conversational Assistant for Fairness Testing

• URL: http://arxiv.org/abs/2512.05428v1
• Date: Fri, 05 Dec 2025 05:04:59 GMT
• Title: Bita: A Conversational Assistant for Fairness Testing
• Abstract summary: Bita is a conversational assistant designed to help software testers detect potential sources of bias.<n>Our work contributes a practical tool that operationalizes fairness testing in a way that is accessible, systematic, and directly applicable to industrial practice.
• Score: 2.198430261120653
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bias in AI systems can lead to unfair and discriminatory outcomes, especially when left untested before deployment. Although fairness testing aims to identify and mitigate such bias, existing tools are often difficult to use, requiring advanced expertise and offering limited support for real-world workflows. To address this, we introduce Bita, a conversational assistant designed to help software testers detect potential sources of bias, evaluate test plans through a fairness lens, and generate fairness-oriented exploratory testing charters. Bita integrates a large language model with retrieval-augmented generation, grounding its responses in curated fairness literature. Our validation demonstrates how Bita supports fairness testing tasks on real-world AI systems, providing structured, reproducible evidence of its utility. In summary, our work contributes a practical tool that operationalizes fairness testing in a way that is accessible, systematic, and directly applicable to industrial practice.

Related papers

• Software Fairness Testing in Practice [0.21427777919040417]
  This study investigates how software professionals test AI-powered systems for fairness through interviews with 22 practitioners working on AI and ML projects.<n>Our findings highlight a significant gap between theoretical fairness concepts and industry practice.<n>Key challenges include data quality and diversity, time constraints, defining effective metrics, and ensuring model interoperability.
  arXiv  Detail & Related papers  (2025-06-20T16:03:02Z)
• Towards Automated Formal Verification of Backend Systems with LLMs [9.66648456498893]
  We propose a novel framework that leverages functional programming and type systems to translate backend code into formal Lean representations.<n>Our pipeline automatically generates theorems that specify the intended behavior of APIs and database operations, and uses LLM-based provers to verify them.<n>We evaluate our method on realistic backend systems and find that it can formally verify over 50% of the test requirements, which suggests that half of a testing engineer's workload can be automated.
  arXiv  Detail & Related papers  (2025-04-13T16:49:37Z)
• Test-Time Visual In-Context Tuning [85.62916644835902]
  Visual in-context learning (VICL) allows the model to rapidly adapt to various tasks with only a handful of prompts and examples.<n>While effective, the existing VICL paradigm exhibits poor generalizability under distribution shifts.<n>We propose test-time Visual In-Context Tuning (VICT), a method that can adapt VICL models on the fly with a single test sample.
  arXiv  Detail & Related papers  (2025-03-27T17:59:52Z)
• Automated Soap Opera Testing Directed by LLMs and Scenario Knowledge: Feasibility, Challenges, and Road Ahead [43.15092098658384]
  Exploratory testing (ET) harnesses tester's knowledge, creativity, and experience to create varying tests that uncover unexpected bugs from the end-user's perspective.<n>We explore the feasibility, challenges and road ahead of automated scenario-based ET (a.k.a soap opera testing)
  arXiv  Detail & Related papers  (2024-12-11T17:57:23Z)
• Which Combination of Test Metrics Can Predict Success of a Software Project? A Case Study in a Year-Long Project Course [1.553083901660282]
  Testing plays an important role in securing the success of a software development project. We investigate whether we can quantify the effects various types of testing have on functional suitability.
  arXiv  Detail & Related papers  (2024-08-22T04:23:51Z)
• Search-Based Fairness Testing: An Overview [4.453735522794044]
  biases in AI systems raise ethical and societal concerns. This paper reviews current research on fairness testing, particularly its application through search-based testing.
  arXiv  Detail & Related papers  (2023-11-10T16:47:56Z)
• Fair Few-shot Learning with Auxiliary Sets [53.30014767684218]
  In many machine learning (ML) tasks, only very few labeled data samples can be collected, which can lead to inferior fairness performance. In this paper, we define the fairness-aware learning task with limited training samples as the emphfair few-shot learning problem. We devise a novel framework that accumulates fairness-aware knowledge across different meta-training tasks and then generalizes the learned knowledge to meta-test tasks.
  arXiv  Detail & Related papers  (2023-08-28T06:31:37Z)
• Fairness-guided Few-shot Prompting for Large Language Models [93.05624064699965]
  In-context learning can suffer from high instability due to variations in training examples, example order, and prompt formats. We introduce a metric to evaluate the predictive bias of a fixed prompt against labels or a given attributes. We propose a novel search strategy based on the greedy search to identify the near-optimal prompt for improving the performance of in-context learning.
  arXiv  Detail & Related papers  (2023-03-23T12:28:25Z)
• Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learning [65.12226891589592]
  This paper proposes a new approach to automated game validation and testing. Our method leverages a data-driven imitation learning technique, which requires little effort and time and no knowledge of machine learning or programming.
  arXiv  Detail & Related papers  (2022-08-15T11:08:44Z)
• Benchopt: Reproducible, efficient and collaborative optimization benchmarks [67.29240500171532]
  Benchopt is a framework to automate, reproduce and publish optimization benchmarks in machine learning. Benchopt simplifies benchmarking for the community by providing an off-the-shelf tool for running, sharing and extending experiments.
  arXiv  Detail & Related papers  (2022-06-27T16:19:24Z)
• Combining Self-Training and Self-Supervised Learning for Unsupervised Disfluency Detection [80.68446022994492]
  In this work, we explore the unsupervised learning paradigm which can potentially work with unlabeled text corpora. Our model builds upon the recent work on Noisy Student Training, a semi-supervised learning approach that extends the idea of self-training.
  arXiv  Detail & Related papers  (2020-10-29T05:29:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.