Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection

• URL: http://arxiv.org/abs/2505.07870v1
• Date: Fri, 09 May 2025 17:48:34 GMT
• Title: Efficient Fairness Testing in Large Language Models: Prioritizing Metamorphic Relations for Bias Detection
• Authors: Suavis Giramata, Madhusudan Srinivasan, Venkat Naidu Gudivada, Upulee Kanewala,
• Abstract summary: Large Language Models (LLMs) are increasingly deployed in various applications, raising critical concerns about fairness and potential biases in their outputs.<n>This paper explores the prioritization of metamorphic relations (MRs) in metamorphic testing as a strategy to efficiently detect fairness issues within LLMs.<n>We apply a sentence diversity-based approach to compute and rank MRs to optimize fault detection.
• Score: 2.1786693199184346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly deployed in various applications, raising critical concerns about fairness and potential biases in their outputs. This paper explores the prioritization of metamorphic relations (MRs) in metamorphic testing as a strategy to efficiently detect fairness issues within LLMs. Given the exponential growth of possible test cases, exhaustive testing is impractical; therefore, prioritizing MRs based on their effectiveness in detecting fairness violations is crucial. We apply a sentence diversity-based approach to compute and rank MRs to optimize fault detection. Experimental results demonstrate that our proposed prioritization approach improves fault detection rates by 22% compared to random prioritization and 12% compared to distance-based prioritization, while reducing the time to the first failure by 15% and 8%, respectively. Furthermore, our approach performs within 5% of fault-based prioritization in effectiveness, while significantly reducing the computational cost associated with fault labeling. These results validate the effectiveness of diversity-based MR prioritization in enhancing fairness testing for LLMs.

Related papers

• Search-based Selection of Metamorphic Relations for Optimized Robustness Testing of Large Language Models [4.278063359062737]
  This paper proposes a search-based approach for optimizing the Metamorphic Relations groups to maximize failure detection and minimize the execution cost.<n>We conducted comparative experiments on the four search algorithms along with a random search, using two major Large Language Models (LLMs)<n>We identified silver bullet MRs for the robustness testing, which demonstrated dominant capabilities in confusing LLMs across different Text-to-Text tasks.
  arXiv  Detail & Related papers  (2025-07-08T01:11:27Z)
• Meta-Fair: AI-Assisted Fairness Testing of Large Language Models [2.9632404823837777]
  Fairness is a core principle in the development of Artificial Intelligence (AI) systems.<n>Current approaches to fairness testing in large language models (LLMs) often rely on manual evaluation, fixed templates, deterministics, and curated datasets.<n>This work aims to lay the groundwork for a novel, automated method for testing fairness in LLMs.
  arXiv  Detail & Related papers  (2025-07-03T11:20:59Z)
• Assessing Judging Bias in Large Reasoning Models: An Empirical Study [99.86300466350013]
  Large Reasoning Models (LRMs) like DeepSeek-R1 and OpenAI-o1 have demonstrated remarkable reasoning capabilities.<n>We present a benchmark comparing judging biases between LLMs and LRMs across both subjective preference-alignment datasets and objective fact-based datasets.
  arXiv  Detail & Related papers  (2025-04-14T07:14:27Z)
• Less is More: Efficient Black-box Attribution via Minimal Interpretable Subset Selection [52.716143424856185]
  We propose LiMA (Less input is More faithful for Attribution), which reformulates the attribution of important regions as an optimization problem for submodular subset selection.<n>LiMA identifies both the most and least important samples while ensuring an optimal attribution boundary that minimizes errors.<n>Our method also outperforms the greedy search in attribution efficiency, being 1.6 times faster.
  arXiv  Detail & Related papers  (2025-04-01T06:58:15Z)
• Optimizing Metamorphic Testing: Prioritizing Relations Through Execution Profile Dissimilarity [2.6749261270690434]
  An oracle determines whether the output of a program for executed test cases is correct. For machine learning programs, such an oracle is often unavailable or impractical to apply. Prioritizing MRs enhances fault detection effectiveness and improves testing efficiency.
  arXiv  Detail & Related papers  (2024-11-14T04:14:30Z)
• An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking [50.81324768683995]
  FIRST is a novel approach that integrates a learning-to-rank objective and leveraging the logits of only the first generated token. We extend the evaluation of FIRST to the TREC Deep Learning datasets (DL19-22), validating its robustness across diverse domains. Our experiments confirm that fast reranking with single-token logits does not compromise out-of-domain reranking quality.
  arXiv  Detail & Related papers  (2024-11-08T12:08:17Z)
• Reward-Augmented Data Enhances Direct Preference Alignment of LLMs [63.32585910975191]
  We introduce reward-conditioned Large Language Models (LLMs) that learn from the entire spectrum of response quality within the dataset.<n>We show that our approach consistently boosts DPO by a considerable margin.<n>Our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion.
  arXiv  Detail & Related papers  (2024-10-10T16:01:51Z)
• FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection [29.20073572683383]
  We propose FAST, a method that boosts existing prioritization methods through guided FeAture SelecTion. FAST is based on the insight that certain features may introduce noise that affects the model's output confidence. It quantifies the importance of each feature for the model's correct predictions, and then dynamically prunes the information from the noisy features.
  arXiv  Detail & Related papers  (2024-09-13T18:13:09Z)
• Electroencephalogram Emotion Recognition via AUC Maximization [0.0]
  Imbalanced datasets pose significant challenges in areas including neuroscience, cognitive science, and medical diagnostics.<n>This study addresses the issue class imbalance, using the Liking' label in the DEAP dataset as an example.
  arXiv  Detail & Related papers  (2024-08-16T19:08:27Z)
• False Correlation Reduction for Offline Reinforcement Learning [115.11954432080749]
  We propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL)
  arXiv  Detail & Related papers  (2021-10-24T15:34:03Z)
• Metamorphic Relation Prioritization for Effective Regression Testing [11.033596835816422]
  We propose approaches to prioritize metamorphic relations (MRs) to improve the efficiency and effectiveness of regression testing. We present two MR prioritization approaches: (1) fault-based and (2) coverage-based. Our results show that fault-based MR prioritization leads to reducing the number of source and follow-up test cases that needs to be executed.
  arXiv  Detail & Related papers  (2021-09-20T19:06:17Z)
• Causal Feature Selection for Algorithmic Fairness [61.767399505764736]
  We consider fairness in the integration component of data management. We propose an approach to identify a sub-collection of features that ensure the fairness of the dataset.
  arXiv  Detail & Related papers  (2020-06-10T20:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.