Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

• URL: http://arxiv.org/abs/2608.18578v2
• Date: Thu, 20 Aug 2026 09:45:51 GMT
• Title: Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs
• Abstract summary: Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate.<n>PTQ is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested.<n>We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models.
• Score: 0.23438564092609357
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests ($p \le 2.6 \times 10^{-6}$) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials, $p = 4.8 \times 10^{-7}$). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget

Related papers

• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Silent Failures in Quantized LLM Reasoning: A Taxonomy-Based Analysis of Hollow Convergence and Failure Mode Shifts [0.0]
  We show that post-training quantization can silently alter how large language models reason even when task accuracy is preserved.<n>We classify 30,000 chain-of-thought outputs from five instruction-tuned LLMs across three quantization precisions.<n>We find that while accuracy is robust across precisions, Hollow Convergence shows a significant size-dependent shift under NF4.
  arXiv  Detail & Related papers  (2026-07-10T21:55:05Z)
• Evaluating Fine-Tuning and Metrics for Neural Decompilation of Dart AOT Binaries [0.0]
  We present a systematic empirical study of fine-tuning effectiveness and validity for neural decompilation.<n>We evaluate six finetuned model variants across three base architectures using three metrics: CodeBLEU, compile@k, and pass@k.<n>Crosslingual interference from Swift training is highly significant at 4B (-2.66 pp, p but statistically indistinguishable from zero at 8B, consistent with the scaling hypothesis)
  arXiv  Detail & Related papers  (2026-07-07T10:36:12Z)
• An Empirical Study of OpenPangu Quantization on Ascend NPUs [43.29474342134475]
  OpenPangu models are attractive targets for private and domestic large-language-model deployment.<n>This paper conducts a controlled empirical study of OpenPangu 1B and 7B models on Huawei Ascend 910B1 NPUs.
  arXiv  Detail & Related papers  (2026-06-19T09:33:11Z)
• Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery [40.94400211806987]
  We show that aggressive 2-bit inference can fail to deliver end-to-end speedup because instability in the generation process inflates total token count.<n>We analyze full reasoning traces of Qwen3 reasoning models across mathematical and commonsense benchmarks.<n>Our results show that extreme low-bit reasoning becomes practical when its failures are treated as controllable generation pathologies.
  arXiv  Detail & Related papers  (2026-06-01T10:04:09Z)
• When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation [9.055086193088083]
  We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models.<n>Mean-bearing perturbations alter final answers more often than presentation perturbations of comparable severity.
  arXiv  Detail & Related papers  (2026-05-25T15:57:11Z)
• ActQuant: Sub-4-bit Action-Guided Quantization for Vision-Language-Action Models [45.62029693245481]
  We introduce ActQuant, an action-guided mixed-precision PTQ framework.<n>ActQuant operates in two stages: (1) an inter-tensor bit allocator that assigns each weight matrix a single bitwidth based on how much it contributes to predicting the agent's actions.<n>We also introduce OmniModel, an agentic conversion pipeline that ports into a native C/C++ with efficient low-bit kernels.
  arXiv  Detail & Related papers  (2026-05-19T19:57:26Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• Quantization Blindspots: How Model Compression Breaks Backdoor Defenses [1.8646269335840449]
  We study how backdoor defenses behave under standard quantization pipelines.<n>We find that INT8 quantization reduces the detection rate of all evaluated defenses to 0% while leaving attack success rates above 99%.<n>Our results expose a mismatch between how defenses are commonly evaluated and how models are actually deployed.
  arXiv  Detail & Related papers  (2025-12-06T02:04:32Z)
• On Jailbreaking Quantized Language Models Through Fault Injection Attacks [3.5902882430875174]
  The safety alignment of Language Models (LMs) is a critical concern, yet their integrity can be challenged by direct parameter manipulation attacks.<n>This paper investigates the efficacy of such attacks for jailbreaking aligned LMs across different quantization schemes.
  arXiv  Detail & Related papers  (2025-07-04T00:48:48Z)
• Fast Quiet-STaR: Thinking Without Thought Tokens [51.79231070632772]
  Fast Quiet STaR is a more efficient reasoning framework that preserves the benefits of token-level reasoning while reducing computational cost.<n>Our method introduces a curriculum learning based training strategy that gradually reduces the number of thought tokens.<n>Experiments on four benchmark datasets with Mistral 7B and Qwen2.5 7B demonstrate that Fast Quiet-STaR consistently outperforms Quiet-STaR in terms of average accuracy.
  arXiv  Detail & Related papers  (2025-05-23T11:14:12Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models [61.474101404805545]
  Diffusion models can generate high-quality images, but as they scale, rising memory demands and higher latency pose deployment challenges.<n>We propose SVDQuant, a new 4-bit quantization paradigm to overcome this limitation.<n>We reduce the memory usage for the 12B FLUX.1 models by 3.5$times$, achieving 3.0$times$ speedup over the 4-bit weight-only quantization (W4A16) baseline.
  arXiv  Detail & Related papers  (2024-11-07T18:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.