Uncertainty Drives Social Bias Changes in Quantized Large Language Models

• URL: http://arxiv.org/abs/2602.06181v1
• Date: Thu, 05 Feb 2026 20:37:26 GMT
• Title: Uncertainty Drives Social Bias Changes in Quantized Large Language Models
• Abstract summary: Post-training quantization reduces the computational cost of large language models.<n>We present the first large-scale study of 50 quantized models evaluated on PostTrainingBiasBench.
• Score: 3.921671869527028
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training quantization reduces the computational cost of large language models but fundamentally alters their social biases in ways that aggregate metrics fail to capture. We present the first large-scale study of 50 quantized models evaluated on PostTrainingBiasBench, a unified benchmark of 13 closed- and open-ended bias datasets. We identify a phenomenon we term quantization-induced masked bias flipping, in which up to 21% of responses flip between biased and unbiased states after quantization, despite showing no change in aggregate bias scores. These flips are strongly driven by model uncertainty, where the responses with high uncertainty are 3-11x more likely to change than the confident ones. Quantization strength amplifies this effect, with 4-bit quantized models exhibiting 4-6x more behavioral changes than 8-bit quantized models. Critically, these changes create asymmetric impacts across demographic groups, where bias can worsen by up to 18.6% for some groups while improving by 14.1% for others, yielding misleadingly neutral aggregate outcomes. Larger models show no consistent robustness advantage, and group-specific shifts vary unpredictably across model families. Our findings demonstrate that compression fundamentally alters bias patterns, requiring crucial post-quantization evaluation and interventions to ensure reliability in practice.

Related papers

• Reliability Scaling Laws for Quantized Large Language Models [68.39028482878265]
  Quantization is a strategy to build capable and resource-efficient large language models (LLMs) by reducing the bitwidth of the parameters.<n>We first conduct a comprehensive reliability evaluation of quantized LLMs consisting of three key components.<n>A reliability peak occurs for 4-bit quantized models, indicating that quantizing moderately sized models offers the best reliability-efficiency trade-off.
  arXiv  Detail & Related papers  (2026-07-12T17:39:09Z)
• Quantization Undoes Alignment: Bias Emergence in Compressed LLMs Across Models and Precision Levels [0.0]
  Large Language Models are routinely compressed via post-training quantization to reduce inference costs and memory footprint for cloud and edge deployment.<n>Existing studies typically compare only two conditions, rely on aggregate bias metrics, and evaluate a single model family.<n>We conduct a controlled empirical study of three instruction-tuned models at five precision levels (BF16 through 3-bit) on 12,148 BBQ bias benchmark items.
  arXiv  Detail & Related papers  (2026-05-02T05:41:47Z)
• Towards a more efficient bias detection in financial language models [5.317864735982288]
  Bias in financial language models is a major obstacle to their adoption in real-world applications.<n>We conduct a large-scale study of bias in five financial language models.<n>We explore cross-model-guided bias detection to identify bias-revealing inputs earlier.
  arXiv  Detail & Related papers  (2026-03-09T11:38:53Z)
• Self-Transparency Failures in Expert-Persona LLMs: A Large-Scale Behavioral Audit [0.0]
  This study examines whether models exhibit self-transparency when assigned professional personas in high-stakes domains.<n>A Financial Advisor persona elicited 30.8% disclosure at the first prompt, while a Neurosurgeon persona elicited only 3.5%.<n>This creates the preconditions for a hypothesized Reverse Gell-Mann Amnesia effect, where appropriate disclosure in some domains leads users to overgeneralize trust to high-stakes contexts.
  arXiv  Detail & Related papers  (2025-11-26T16:41:49Z)
• Causal Understanding by LLMs: The Role of Uncertainty [43.87879175532034]
  Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
  arXiv  Detail & Related papers  (2025-09-24T13:06:35Z)
• How Quantization Shapes Bias in Large Language Models [61.40435736418359]
  We focus on weight and activation quantization strategies and examine their effects across a broad range of bias types.<n>We employ both probabilistic and generated text-based metrics across nine benchmarks and evaluate models varying in architecture family and reasoning ability.
  arXiv  Detail & Related papers  (2025-08-25T14:48:26Z)
• Improving Group Robustness on Spurious Correlation via Evidential Alignment [26.544938760265136]
  Deep neural networks often learn and rely on spurious correlations, i.e., superficial associations between non-causal features and the targets.<n>Existing methods typically mitigate this issue by using external group annotations or auxiliary deterministic models.<n>We propose Evidential Alignment, a novel framework that leverages uncertainty quantification to understand the behavior of the biased models.
  arXiv  Detail & Related papers  (2025-06-12T22:47:21Z)
• Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models [48.98109982725689]
  We conduct the first systematic study on quantized reasoning models.<n>Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths.<n>We identify model size, model origin, and task difficulty as critical determinants of performance.
  arXiv  Detail & Related papers  (2025-04-07T08:22:45Z)
• You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models [15.067035577724502]
  We study the impact of a standard practice in compressing foundation vision-language models - quantization - on the models' ability to produce socially-fair outputs. We find no consistent change in bias magnitude or direction across a population of compressed models due to quantization.
  arXiv  Detail & Related papers  (2024-10-26T20:31:41Z)
• Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant [11.832907585157638]
  Quantization has gained attention as a promising solution for the cost-effective deployment of large and small language models.<n>We conduct a comprehensive evaluation of instruction-tuned models spanning 1B to 405B parameters, applying four quantization methods across 13 datasets.
  arXiv  Detail & Related papers  (2024-09-17T10:31:37Z)
• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• RobustMQ: Benchmarking Robustness of Quantized Models [54.15661421492865]
  Quantization is an essential technique for deploying deep neural networks (DNNs) on devices with limited resources. We thoroughly evaluated the robustness of quantized models against various noises (adrial attacks, natural corruptions, and systematic noises) on ImageNet. Our research contributes to advancing the robust quantization of models and their deployment in real-world scenarios.
  arXiv  Detail & Related papers  (2023-08-04T14:37:12Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures [93.17009514112702]
  Pruning, setting a significant subset of the parameters of a neural network to zero, is one of the most popular methods of model compression. Despite existing evidence for this phenomenon, the relationship between neural network pruning and induced bias is not well-understood.
  arXiv  Detail & Related papers  (2023-04-25T07:42:06Z)
• Monotonicity and Double Descent in Uncertainty Estimation with Gaussian Processes [52.92110730286403]
  It is commonly believed that the marginal likelihood should be reminiscent of cross-validation metrics and that both should deteriorate with larger input dimensions. We prove that by tuning hyper parameters, the performance, as measured by the marginal likelihood, improves monotonically with the input dimension. We also prove that cross-validation metrics exhibit qualitatively different behavior that is characteristic of double descent.
  arXiv  Detail & Related papers  (2022-10-14T08:09:33Z)
• Flexible Model Aggregation for Quantile Regression [92.63075261170302]
  Quantile regression is a fundamental problem in statistical learning motivated by a need to quantify uncertainty in predictions. We investigate methods for aggregating any number of conditional quantile models. All of the models we consider in this paper can be fit using modern deep learning toolkits.
  arXiv  Detail & Related papers  (2021-02-26T23:21:16Z)
• Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures [62.562760228942054]
  Existing approaches to improve robustness against dataset biases mostly focus on changing the training objective. We propose to augment the input sentences in the training data with their corresponding predicate-argument structures. We show that without targeting a specific bias, our sentence augmentation improves the robustness of transformer models against multiple biases.
  arXiv  Detail & Related papers  (2020-10-23T16:22:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.