QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

• URL: http://arxiv.org/abs/2607.21063v1
• Date: Thu, 23 Jul 2026 08:56:11 GMT
• Title: QuantiBias: Benchmarking Quantization-Induced Bias in LLMs
• Abstract summary: We find that almost every large language model that reaches a broad audience is quantized.<n>Our benchmark pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation.<n>A quantized build must be re-evaluated for open-ended bias, not only on the short-form safeguards it already passes.
• Score: 8.970269049715933
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is assumed harmless and its safety is rarely re-checked. We find its principal side effect is increased bias that standard safety evaluation misses. Holding the model, its training, and the prompts fixed, a quantized model still refuses harmful requests, still avoids over-refusing benign prompts, and still selects the unbiased multiple-choice answer. Yet asked an open-ended question, the same model volunteers stereotypes in all eight languages we probe, in roughly one in four open-ended answers under an independent judge (~24% to ~27% across the compression ladder): it passes every standard check and still reaches users measurably more biased. The selective gap is a robust finding; whether open-ended bias further increases with compression is less certain, sensitive to the judge that scores it. We address both with \textbf{QuantiBias}, a benchmark that pairs a generative, multilingual stereotype probe with the refusal and multiple-choice controls that isolate open-ended generation, contrasts each build with and without reasoning, and rates the content severity of what it generates. Across two backbone models (Qwen and Gemma), a five-family screen, and eight benchmarks, quantizers allocate their extra precision by capability data that carries no bias-prevention signal, and reasoning before answering roughly halves the effect on some families while doing nothing on others. A quantized build must be re-evaluated for open-ended bias, not only on the short-form safeguards it already passes.

Related papers

• FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect [69.8983512616053]
  Large language models often change their responses to subtle rephrasings that align with an implied stance by users.<n>This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts.<n>We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations.
  arXiv  Detail & Related papers  (2026-09-07T12:56:44Z)
• Learning When to Trust via Selective Context Preference Optimization [34.54211638111961]
  Language models increasingly condition their answers on external signals.<n>A single misleading one can turn a correct answer wrong.<n>We recast the problem as selective trust.
  arXiv  Detail & Related papers  (2026-08-06T17:59:58Z)
• BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence [22.315546054051143]
  Bias audits of large language models now operate within governance frameworks such as the EU AI Act.<n>We introduce textscBiAxisAudit, a protocol that reports each bias score together with a reliability estimate on two axes.
  arXiv  Detail & Related papers  (2026-05-09T16:26:49Z)
• Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation [0.0]
  Large language models pick up social biases from the data they are trained on and carry those biases into downstream applications.<n>In this paper we take a different route: we debias the model at decoding time, treating bias mitigation as a structured search over candidate tokens without ever touching model weights.<n>We design three schemes of increasing sophistication (Best-of-N selection, Sequential critique-and-revise, and Constitutional self-audit) and evaluate them on four models.
  arXiv  Detail & Related papers  (2026-05-04T08:51:34Z)
• ImplicitBBQ: Benchmarking Implicit Bias in Large Language Models through Characteristic Based Cues [8.68634179512072]
  Implicit bias in ambiguous contexts is over six times higher than explicit bias in open weight models.<n>Even few-shot prompting, which reduces implicit bias by 84%, leaves caste bias at four times the level of any other dimension.<n>We release our code and dataset for model providers and researchers to benchmark potential mitigation techniques.
  arXiv  Detail & Related papers  (2026-04-02T11:43:10Z)
• Adaptive Generation of Bias-Eliciting Questions for LLMs [18.608477560948003]
  Large language models (LLMs) are now widely deployed in user-facing applications, reaching hundreds of millions worldwide.<n>We introduce a counterfactual bias evaluation framework that automatically generates realistic, open-ended questions over sensitive attributes such as sex, race, or religion.<n>We also capture distinct response dimensions that are increasingly relevant in user interactions, such as asymmetric refusals and explicit acknowledgment of bias.
  arXiv  Detail & Related papers  (2025-10-14T13:08:10Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• Open-DeBias: Toward Mitigating Open-Set Bias in Language Models [6.958242323649994]
  We tackle the novel problem of open-set bias detection and mitigation in text-based question answering tasks.<n>We introduce OpenBiasBench, a benchmark designed to evaluate biases across a wide range of categories and subgroups.<n>We also propose Open-DeBias, a novel, data-efficient, and parameter-efficient debiasing method.
  arXiv  Detail & Related papers  (2025-09-28T11:08:39Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• OpenBias: Open-set Bias Detection in Text-to-Image Generative Models [108.2219657433884]
  We tackle the challenge of open-set bias detection in text-to-image generative models presenting OpenBias. OpenBias identifies and quantifies the severity of biases agnostically, without access to any precompiled set. We study the behavior of Stable Diffusion 1.5, 2, and XL emphasizing new biases, never investigated before.
  arXiv  Detail & Related papers  (2024-04-11T17:59:56Z)
• Mitigating Bias for Question Answering Models by Tracking Bias Influence [84.66462028537475]
  We propose BMBI, an approach to mitigate the bias of multiple-choice QA models. Based on the intuition that a model would lean to be more biased if it learns from a biased example, we measure the bias level of a query instance. We show that our method could be applied to multiple QA formulations across multiple bias categories.
  arXiv  Detail & Related papers  (2023-10-13T00:49:09Z)
• UnQovering Stereotyping Biases via Underspecified Questions [68.81749777034409]
  We present UNQOVER, a framework to probe and quantify biases through underspecified questions. We show that a naive use of model scores can lead to incorrect bias estimates due to two forms of reasoning errors. We use this metric to analyze four important classes of stereotypes: gender, nationality, ethnicity, and religion.
  arXiv  Detail & Related papers  (2020-10-06T01:49:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.