Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

• URL: http://arxiv.org/abs/2609.01962v1
• Date: Wed, 02 Sep 2026 00:42:54 GMT
• Title: Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment
• Abstract summary: Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior.<n>We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B- parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA.<n>We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior. We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B-parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA. The experiment is weight-only: activations remain at 16-bit precision, so ILA-AMP is omitted. We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior. The final conversion uses 1.641 effective bits per weight for quantized linear weights, with 81.62% of model parameters targeted. Across ten scored capability comparisons, accuracy falls from 64.5% to 54.7%. Degradation is uneven: BoolQ retains 84.6% chance-corrected teacher performance, while ARC-Challenge retains 43.8%. Perplexity rises from 13.639 to 18.748 on WikiText-2, 24.700 to 31.992 on PTB, and 19.831 to 28.966 on C4. A subsequent packing run preserves the ternary planes and scales, reducing reported model size from 8.29 GiB to 3.96 GiB with essentially unchanged perplexity. A separate third-party packing attempt was lossy and is excluded from the primary artifact claim. The packed artifact has not been benchmarked end-to-end for task accuracy or generation throughput. A preliminary Triton GEMV microbenchmark is 4.6x slower than FP16 cuBLAS on one tested shape. We therefore do not claim that compression alone yields faster inference.

Related papers

• Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution [0.0]
  We study a scale-up of an aggressive post-training conversion pipeline from Qwen3-4B to Qwen3-8B.<n>The conversion uses KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style error compensation in a weight-only A16 configuration.
  arXiv  Detail & Related papers  (2026-09-08T06:21:44Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Capability-Stratified Degradation in Ternary Language Models [0.0]
  Ternary language models restrict weights to $-1,0,+1$, approaching the limit of $log 3 approx 1.585$ bits/weight.<n>We convert Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT)<n>The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning.
  arXiv  Detail & Related papers  (2026-08-28T19:22:38Z)
• MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models [49.47300047353726]
  We introduce MemToC, a benchmark for post-tool-return arbitration with executable tools.<n>MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions.<n>Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers.
  arXiv  Detail & Related papers  (2026-08-26T18:22:03Z)
• Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models [0.0]
  We introduce Variable Bit-width Quantization (VBQ)<n>Training-time method in which each contiguous group of 64 weights learns its own resolution from 1,2,4,8 bits.<n>As quality-per-byte, VBQ is 3.9-8.4x more efficient than FP16.
  arXiv  Detail & Related papers  (2026-07-03T02:42:04Z)
• Output-Space Allocation Costs for Calibration-Guided LLM Compression: An Empirical Study [19.413175066072302]
  Training-free compression methods for large language models (LLMs) often use calibration data to guide compression decisions.<n>ROCKET, a recent method combining sparse-dictionary factorization with multi-choice knapsack problem (MCKP) allocation, derives its per-layer factorization from an output reconstruction objective.<n>We investigate whether aligning the allocation cost with the output-space objective improves compressed model fidelity.
  arXiv  Detail & Related papers  (2026-06-26T07:15:17Z)
• BitCal-TTS: Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models [0.0]
  BitCal-TTS is a lightweight runtime controller for greedy 4-bit inference.<n>It integrates with standard Hugging Face 4-bit inference using forward hooks for logits and last-layer hidden states.<n>On evaluation shards of GSM8K with Qwen2.5 Instruct models, BitCal-TTS improves exact-match accuracy over a non-bit-aware adaptive baseline.
  arXiv  Detail & Related papers  (2026-05-07T01:10:34Z)
• Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression [55.323397702682506]
  Post-training quantization (PTQ) reduces a model's memory footprint by mapping full precision weights into low bit weights without costly retraining.<n>We develop a new mixed-precision PTQ approach, Task-Circuit Quantization (TaCQ), that draws parallels to automated circuit discovery.
  arXiv  Detail & Related papers  (2025-04-10T02:19:03Z)
• QLoRA: Efficient Finetuning of Quantized LLMs [66.58009990713134]
  We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU. QLoRA backpropagates through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters(LoRA) Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark.
  arXiv  Detail & Related papers  (2023-05-23T17:50:33Z)
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance [68.8204255655161]
  We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline.
  arXiv  Detail & Related papers  (2023-01-31T02:46:57Z)
• The case for 4-bit precision: k-bit Inference Scaling Laws [75.4335600212427]
  Quantization methods reduce the number of bits required to represent each parameter in a model. The final model size depends on both the number of parameters of the original model and the rate of compression. We run more than 35,000 zero-shot experiments with 16-bit inputs and k-bit parameters to examine which quantization methods improve scaling for 3 to 8-bit precision.
  arXiv  Detail & Related papers  (2022-12-19T18:48:33Z)
• Efficient Speech Representation Learning with Low-Bit Quantization [32.75829498841329]
  We apply and investigate recent quantization techniques on speech representation learning models. With aggressive quantization to 1 bit, we achieved 86.32% storage reduction (184.42 -> 25.23), 88% estimated runtime reduction (1.00 -> 0.12) with increased word error rate (7.06 -> 15.96). In comparison with DistillHuBERT which also aims for model compression, the 2-bit configuration yielded slightly smaller storage (35.84 vs. 46.98), better word error rate (12.68 vs. 13.37) and more efficient runtime estimated (0.15 vs. 0.73)
  arXiv  Detail & Related papers  (2022-12-14T06:09:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.