Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution

• URL: http://arxiv.org/abs/2609.09240v1
• Date: Tue, 08 Sep 2026 06:21:44 GMT
• Title: Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution
• Abstract summary: We study a scale-up of an aggressive post-training conversion pipeline from Qwen3-4B to Qwen3-8B.<n>The conversion uses KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style error compensation in a weight-only A16 configuration.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ultra-low-bit language models promise reductions in storage and memory traffic, but a nominal "1.58-bit" label does not specify the deployed representation or its execution cost. We study a scale-up of an aggressive post-training conversion pipeline from Qwen3-4B to Qwen3-8B. The conversion uses KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style error compensation in a weight-only A16 configuration. We do not claim these algorithms as new. Our contribution is the end-to-end scale-up characterisation: an external reproduction gate, matched 4B/8B capability analysis, cross-corpus perplexity, effective-bit accounting, lossless lattice-aware packing, and direct packed execution. The 8B model reaches a three-corpus perplexity ratio of 1.361x, with WikiText-2, C4, and PTB ratios of 1.318x, 1.393x, and 1.371x. On eight zero-shot tasks at n = 500, mean accuracy is 64.6% versus 72.4% for FP16, corresponding to 78.5% chance-corrected retention and a 7.8-point absolute cost. The matched 4B run retains 69.6%, yielding an 8.9-point 8B advantage. The packed checkpoint is 8.24 GiB and preserves the recorded perplexity to measurement precision. Direct packed execution reaches 15.52 tokens/s in 7.35 GiB, while a preliminary packed GEMV remains slower than FP16 cuBLAS. The result is a validated scale-up baseline: model size improves robustness to aggressive post-training discretisation, actual serialisation is solved for the measured artefact, and direct execution is feasible, while broader seeds, calibration distributions, and kernel optimisation remain open.

Related papers

• Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment [0.0]
  Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior.<n>We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B- parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA.<n>We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior.
  arXiv  Detail & Related papers  (2026-09-02T00:42:54Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Variable Bit-width Quantization: Learning Per-Group Precision for "Bigger-but-Smaller" Language Models [0.0]
  We introduce Variable Bit-width Quantization (VBQ)<n>Training-time method in which each contiguous group of 64 weights learns its own resolution from 1,2,4,8 bits.<n>As quality-per-byte, VBQ is 3.9-8.4x more efficient than FP16.
  arXiv  Detail & Related papers  (2026-07-03T02:42:04Z)
• BitCal-TTS: Bit-Calibrated Test-Time Scaling for Quantized Reasoning Models [0.0]
  BitCal-TTS is a lightweight runtime controller for greedy 4-bit inference.<n>It integrates with standard Hugging Face 4-bit inference using forward hooks for logits and last-layer hidden states.<n>On evaluation shards of GSM8K with Qwen2.5 Instruct models, BitCal-TTS improves exact-match accuracy over a non-bit-aware adaptive baseline.
  arXiv  Detail & Related papers  (2026-05-07T01:10:34Z)
• FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression [5.385515135626162]
  We present FASQ (Flexible Accelerated Subspace Quantization), a calibration-free framework that applies product quantization to large language models.<n>On Meta-Llama-3-8B, FASQ surpasses 4-bit GPTQ and AWQ in accuracy (67-42% model size) at 37-42% model size.
  arXiv  Detail & Related papers  (2026-04-22T20:03:36Z)
• RAMP: Reinforcement Adaptive Mixed Precision Quantization for Efficient On Device LLM Inference [1.1100764382749708]
  RAMP (Reinforcement Adaptive Mixed Precision) learns per layer bit width assignments to minimize perplexity under a global bit budget.<n>On Llama 2 7B, RAMP achieves 5.54 perplexity at 3.68GB (3.65 effective bits), outperforming uniform 4 bit AWQ (5.60 at 3.90 GB) and GPTQ by 6% in size and 1% to3% in quality.
  arXiv  Detail & Related papers  (2026-03-18T16:16:28Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization [73.60493264901359]
  We present a unified framework for rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings.<n>We show that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off.<n>Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.
  arXiv  Detail & Related papers  (2025-02-04T18:59:26Z)
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance [68.8204255655161]
  We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline.
  arXiv  Detail & Related papers  (2023-01-31T02:46:57Z)
• The case for 4-bit precision: k-bit Inference Scaling Laws [75.4335600212427]
  Quantization methods reduce the number of bits required to represent each parameter in a model. The final model size depends on both the number of parameters of the original model and the rate of compression. We run more than 35,000 zero-shot experiments with 16-bit inputs and k-bit parameters to examine which quantization methods improve scaling for 3 to 8-bit precision.
  arXiv  Detail & Related papers  (2022-12-19T18:48:33Z)
• Combined Scaling for Zero-shot Transfer Learning [146.0851484769142]
  We present a combined scaling method - named BASIC - that achieves 85.7% top-1 accuracy on the ImageNet ILSVRC-2012 validation set. This accuracy surpasses best published similar models - CLIP and ALIGN - by 9.3%. Our model also shows significant improvements in robustness benchmarks.
  arXiv  Detail & Related papers  (2021-11-19T05:25:46Z)
• Training with Quantization Noise for Extreme Model Compression [57.51832088938618]
  We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work beyond int8 fixed-point quantization with extreme compression methods.
  arXiv  Detail & Related papers  (2020-04-15T20:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.