Capability-Stratified Degradation in Ternary Language Models

• URL: http://arxiv.org/abs/2608.28809v1
• Date: Fri, 28 Aug 2026 19:22:38 GMT
• Title: Capability-Stratified Degradation in Ternary Language Models
• Abstract summary: Ternary language models restrict weights to $-1,0,+1$, approaching the limit of $log 3 approx 1.585$ bits/weight.<n>We convert Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT)<n>The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Extreme low-bit inference offers a route toward smaller models and constrained deployment. Ternary language models restrict weights to $\{-1,0,+1\}$, approaching the limit of $\log_2 3 \approx 1.585$ bits/weight. The practical question for a pretrained model is not simply whether weights can be quantised but which capabilities survive and whether it remains useful for adaptation. We explore this by converting Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT). The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning. The evidence shows non-uniform degradation. A linear probe recovers 43.76% of MMLU answers from the full-precision teacher's representations but only 26.19% from Cloe (near chance), indicating specialist factual information is lost. However, Cloe retains measurable performance on ten tasks, averaging 77.1% of teacher performance. Crucially, fine-tuning raises Cloe to 89.8% on SST-2 (95.6% of the matched teacher) and reaches 79.4% teacher retention on XSum. We attribute degradation to a combination of quantisation-induced information loss and incomplete recovery due to the limited QAT budget. We also highlight an evaluation pitfall: standard answer-letter scoring failed (Cloe emitted "A" on 98.6% of MMLU questions), necessitating continuation scoring. Ultimately, ternary conversion is unsuitable as a drop-in general replacement yet remains valuable as a compact substrate for task-specific models.

Related papers

• Safe Error Correction for Language Models: Frozen-Base Adjustment with Capability Preservation [0.0]
  We propose a lightweight logit-level correction module atop a frozen language model.<n>CRN v2 corrects 53.3% of base-model errors while showing no degradation on tested capability benchmarks.<n>None of the alternative configurations we tested exceeded the rank-128 logit result.
  arXiv  Detail & Related papers  (2026-09-14T18:00:15Z)
• Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment [0.0]
  Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior.<n>We study an end-to-end post-training conversion of Qwen, an instruction-tuned 4B- parameter model, using KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style error compensation from TWLA.<n>We evaluate effective bit accounting, task capability retention, perplexity, calibration sensitivity, checkpoint composition, and deployment behavior.
  arXiv  Detail & Related papers  (2026-09-02T00:42:54Z)
• Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model [0.0]
  We convert 21 of 28 full-attention layers of Qwen3-0.6B-Base into KDA (Kimi Delta Attention)<n>After surgery, hidden-state alignment and end-to-end KL distillation drive the student close to its teacher in perplexity.<n>We release code, weights, recipes, and the full audit trail, and distill the engineering lessons.
  arXiv  Detail & Related papers  (2026-08-03T08:54:18Z)
• Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition [0.0]
  We build a Chain-of-Thought training corpus through a dual-agent framework.<n>The dataset is used to fine-tune the student model with Low-Rank Adaptation (LoRA) on Apple Silicon hardware.
  arXiv  Detail & Related papers  (2026-06-30T02:34:45Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Fast Unlearning at Scale via Margin Self-Correction [52.46927918952516]
  Language-model unlearning updates a trained model to behave as if it had not seen selected training examples.<n>MASC achieves a competitive forget-retain trade-off at a fraction of the computational cost of existing baselines.
  arXiv  Detail & Related papers  (2026-06-01T21:49:54Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning [0.0]
  We study prompt engineering for formal mathematical reasoning in the context of the SAIR Equational Theories Stage 1 competition.<n>The task requires deciding whether one equational law implies another over all magmas.<n>Over five weeks, we designed, tested, and analyzed more than 40 prompt variants.
  arXiv  Detail & Related papers  (2026-04-20T22:55:23Z)
• One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness [12.183451602438753]
  We show that simple lexical constraints (banning a single punctuation character or common word) cause instruction-tuned LLMs to collapse their responses.<n>We show that base models show no systematic collapse under identical constraints, with effects that are small, noisy, and bidirectional.
  arXiv  Detail & Related papers  (2026-04-14T17:40:01Z)
• Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning [65.2421542320293]
  Reasoning abilities are crucial components of general intelligence.<n>Recent advances by proprietary companies, such as o-series models of OpenAI, have made remarkable progress on reasoning tasks.<n>This paper proposes a new RL framework, termed OREAL, to pursue the performance limit that can be achieved through textbfOutcome textbfREwtextbfArd-based reinforcement textbfLearning for mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2025-02-10T18:57:29Z)
• LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ [0.0]
  Large Language Models (LLMs) often struggle with tasks requiring mathematical reasoning, particularly multiple-choice questions (MCQs) We developed LLaMa-SciQ to assist college students in solving and understanding MCQs in STEM fields. For mathematical reasoning, LLaMa-SciQ achieved 74.5% accuracy on the GSM8k dataset and 30% on the MATH dataset.
  arXiv  Detail & Related papers  (2024-09-25T09:41:46Z)
• Common 7B Language Models Already Possess Strong Math Capabilities [61.61442513067561]
  This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities. The potential for extensive scaling is constrained by the scarcity of publicly available math questions.
  arXiv  Detail & Related papers  (2024-03-07T18:00:40Z)
• QLoRA: Efficient Finetuning of Quantized LLMs [66.58009990713134]
  We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU. QLoRA backpropagates through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters(LoRA) Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark.
  arXiv  Detail & Related papers  (2023-05-23T17:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.