An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation

• URL: http://arxiv.org/abs/2512.02689v1
• Date: Tue, 02 Dec 2025 12:18:48 GMT
• Title: An Empirical Survey of Model Merging Algorithms for Social Bias Mitigation
• Authors: Daiki Shirafuji, Tatsuhiko Saito, Yasutomo Kimura,
• Abstract summary: Large language models (LLMs) are known to inherit and even amplify societal biases present in their pre-training corpora.<n>We empirically survey seven algorithms: Linear, Karcher Mean, SLERP, NuSLERP, TIES, DELLA, and Nearswap, applying 13 open weight models in the GPT, LLaMA, and Qwen families.<n>We find a trade-off between bias reduction and downstream performance: methods achieving greater bias mitigation degrade accuracy.
• Score: 0.9430947207126281
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large language models (LLMs) are known to inherit and even amplify societal biases present in their pre-training corpora, threatening fairness and social trust. To address this issue, recent work has explored ``editing'' LLM parameters to mitigate social bias with model merging approaches; however, there is no empirical comparison. In this work, we empirically survey seven algorithms: Linear, Karcher Mean, SLERP, NuSLERP, TIES, DELLA, and Nearswap, applying 13 open weight models in the GPT, LLaMA, and Qwen families. We perform a comprehensive evaluation using three bias datasets (BBQ, BOLD, and HONEST) and measure the impact of these techniques on LLM performance in downstream tasks of the SuperGLUE benchmark. We find a trade-off between bias reduction and downstream performance: methods achieving greater bias mitigation degrade accuracy, particularly on tasks requiring reading comprehension and commonsense and causal reasoning. Among the merging algorithms, Linear, SLERP, and Nearswap consistently reduce bias while maintaining overall performance, with SLERP at moderate interpolation weights emerging as the most balanced choice. These results highlight the potential of model merging algorithms for bias mitigation, while indicating that excessive debiasing or inappropriate merging methods may lead to the degradation of important linguistic abilities.

Related papers

• Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation [89.52571224447111]
  Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization.<n>We provide an LLM judge bias analysis as a function of overlap with human-written responses in the domain of summarization.
  arXiv  Detail & Related papers  (2026-02-07T19:39:28Z)
• Feature-Wise Mixing for Mitigating Contextual Bias in Predictive Supervised Learning [0.0]
  This paper introduces a feature-wise mixing framework to mitigate contextual bias.<n>It was done by redistributing feature representations across multiple contextual datasets.<n>It achieved an average bias reduction of 43.35% and a statistically significant decrease in mean squared error.
  arXiv  Detail & Related papers  (2025-06-28T23:12:59Z)
• Relative Bias: A Comparative Framework for Quantifying Bias in LLMs [29.112649816695203]
  Relative Bias is a method designed to assess how an LLM's behavior deviates from other LLMs within a specified target domain.<n>We introduce two complementary methodologies: (1) Embedding Transformation analysis, which captures relative bias patterns through sentence representations over the embedding space, and (2) LLM-as-a-Judge, which employs a language model to evaluate outputs comparatively.<n>Applying our framework to several case studies on bias and alignment scenarios following by statistical tests for validation, we find strong alignment between the two scoring methods.
  arXiv  Detail & Related papers  (2025-05-22T01:59:54Z)
• Attention Pruning: Automated Fairness Repair of Language Models via Surrogate Simulated Annealing [14.432850893209817]
  This paper introduces Attention Pruning, a fairness-aware simulated annealing approach to prune attention heads in large language models (LLMs)<n>Our experiments show that Attention Pruning achieves up to $40%$ reduction in gender bias and outperforms the state-of-the-art bias mitigation strategies.
  arXiv  Detail & Related papers  (2025-03-20T03:02:32Z)
• Preference Leakage: A Contamination Problem in LLM-as-a-judge [69.96778498636071]
  Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods.<n>In this work, we expose preference leakage, a contamination problem in LLM-as-a-judge caused by the relatedness between the synthetic data generators and LLM-based evaluators.
  arXiv  Detail & Related papers  (2025-02-03T17:13:03Z)
• A Multi-LLM Debiasing Framework [85.17156744155915]
  Large Language Models (LLMs) are powerful tools with the potential to benefit society immensely, yet, they have demonstrated biases that perpetuate societal inequalities. Recent research has shown a growing interest in multi-LLM approaches, which have been demonstrated to be effective in improving the quality of reasoning. We propose a novel multi-LLM debiasing framework aimed at reducing bias in LLMs.
  arXiv  Detail & Related papers  (2024-09-20T20:24:50Z)
• Identifying and Mitigating Social Bias Knowledge in Language Models [52.52955281662332]
  We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.<n>FAST surpasses state-of-the-art baselines with superior debiasing performance.<n>This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
  arXiv  Detail & Related papers  (2024-08-07T17:14:58Z)
• Optimizing Class-Level Probability Reweighting Coefficients for Equitable Prompting Accuracy [12.287692969438169]
  LLMs often uncover biases from pre-training data's statistical regularities.<n>This leads to persistent, uneven class accuracy in classification and QA.<n>We develop a post-hoc probability reweighting method that directly optimize for non-differentiable performance-driven metrics.
  arXiv  Detail & Related papers  (2024-05-13T10:30:33Z)
• Marginal Debiased Network for Fair Visual Recognition [59.05212866862219]
  We propose a novel marginal debiased network (MDN) to learn debiased representations. Our MDN can achieve a remarkable performance on under-represented samples.
  arXiv  Detail & Related papers  (2024-01-04T08:57:09Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing [33.25539075550122]
  We extend an existing bias benchmark for NLI using a combination of LM-generated lexical variations, adversarial filtering, and human validation. We show that BBNLI-next reduces the accuracy of state-of-the-art NLI models from 95.3% to a strikingly low 57.5%. We propose bias measures that take into account both bias and model brittleness.
  arXiv  Detail & Related papers  (2023-05-22T01:02:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.