Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression

• URL: http://arxiv.org/abs/2608.24070v1
• Date: Tue, 25 Aug 2026 05:09:07 GMT
• Title: Compression Trinity: Exploring Sparsity, Quantization, and Low-Rank Approximations for LLM Compression
• Abstract summary: This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce inversion, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy.<n>Results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.
• Score: 3.8834605840347654
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prohibitive computational and environmental costs impede the scalable deployment of Large Language Models (LLMs). Traditional compression techniques (sparsity, quantization, low-rank approximations) are typically applied in isolation, and each hits an accuracy-efficiency wall. This thesis proposes the "Compression Trinity," a unified framework that applies the three pillars jointly: sparsity to reduce computation, quantization to minimize memory bandwidth, and low-rank approximations to recover accuracy. To accelerate pretraining, we apply the Trinity to the optimizer and model architecture. MKOR approximates curvature via block-diagonal sparsity and low-rank inversion, maintaining numerical stability for quantized states; it reduces curvature update complexity from $O(d^3)$ to $O(d^2)$ and accelerates convergence by up to 1.85x over KFAC. SLoPe accelerates training by up to 1.25x via a double-pruned backward pass for N:M sparsity, using low-rank "lazy" adapters in the final 1% of training to recover accuracy. For post-training compression, OPTIMA stabilizes static masks in a zero-training regime by formulating weight reconstruction as globally optimal column-wise quadratic programs, improving zero-shot accuracy by up to 3.97%. Given a fine-tuning budget, PATCH breaks the ceiling of static masks by learning a dynamic hybrid sparsity ratio between 0% and 50%, yielding up to 1.38x speedups. Finally, SLiM realizes the full Compression Trinity in one shot, using mathematically derived low-rank adapters to recover information lost to quantization and sparsity, improving accuracy by up to 5.66% over state-of-the-art methods and outperforming uncompressed dense models at equal parameter budgets by 0.6%. Together, these results show that jointly applying the Compression Trinity is essential for efficient, scalable, high-performance LLMs.

Related papers

• Per-Matrix Optimality Is Not Enough: Three-Level Optimization for Low-Rank LLM Compression [8.890261387138215]
  We introduce a three-level chain that widens optimization scope from individual matrices to Transformer blocks to the full model.<n>On LLaMA-7B at 60% compression, the chain reduces WikiText-2 perplexity from 42.1 to 19.1 to 11.4.<n> skipping the block-level stage becomes competitive, revealing an offline compute--data trade-off.
  arXiv  Detail & Related papers  (2026-09-14T16:36:46Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization [15.401450705732232]
  Ternarization has emerged as a promising compression technique.<n>Existing methods struggle with heavy-tailed activation distributions.<n>We propose TWLA, a framework that achieves 1.58-bit weight compression and 4-bit activation quantization.
  arXiv  Detail & Related papers  (2026-06-11T08:37:20Z)
• QuantSR+: Pushing the Limit of Quantized Image Super-Resolution Networks [63.04549067734431]
  Low-bit quantization is widely used to compress super-resolution (SR) models.<n>However, when SR models are pushed to ultra-low precision (2-4 bits), performance can drop sharply.<n>We propose QuantSR+, a unified framework that improves quantization operators, network design, and training optimization.
  arXiv  Detail & Related papers  (2026-05-21T11:38:52Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• OPTIMA: Optimal One-shot Pruning for LLMs via Quadratic Programming Reconstruction [12.653025902977001]
  Post-training model pruning is a promising solution, yet it faces a trade-off that zero weights are fast but degrade accuracy.<n>One-shot methods such as SparseGPT offer a practical trade-off in optimality by applying efficient, approximate weight updates.<n>We introduce OPTIMA, a practical one-shot post-training pruning method that balances accuracy and scalability.
  arXiv  Detail & Related papers  (2025-12-15T20:41:29Z)
• EcoSpa: Efficient Transformer Training with Coupled Sparsity [79.5008521101473]
  Transformers have become the backbone of modern AI, yet their high computational demands pose critical system challenges.<n>We introduce EcoSpa, an efficient structured sparse training method that jointly evaluates and sparsifies coupled weight matrix pairs.
  arXiv  Detail & Related papers  (2025-11-09T11:23:43Z)
• Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression [57.71917274869577]
  UltraDelta is a data-free delta compression pipeline that achieves both ultra-high compression and strong performance.<n>UltraDelta is designed to minimize redundancy, maximize information, and stabilize performance across inter-layer, intra-layer, and global dimensions.
  arXiv  Detail & Related papers  (2025-05-19T10:37:22Z)
• Lillama: Large Language Models Compression via Low-Rank Feature Distillation [8.090496457850852]
  Lillama is a compression method that distills activations with low-rank weights.<n>It compresses Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance.<n>It generalizes well to non-transformer architectures, compressing Mamba-3B by 20% while maintaining 99% performance.
  arXiv  Detail & Related papers  (2024-12-21T18:04:01Z)
• EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation [84.70637613266835]
  EoRA is a fine-tuning-free method that augments compressed Large Language Models with low-rank matrices.<n>EoRA consistently outperforms prior training-free low rank methods in recovering the accuracy of compressed LLMs.
  arXiv  Detail & Related papers  (2024-10-28T17:59:03Z)
• SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression [7.6131620435684875]
  SLIM is a new one-shot compression framework that holistically integrates hardware-friendly quantization, sparsity, and low-rank approximation.<n>SLIM improves model accuracy by up to 5.66% (LLaMA-2-7B) for 2:4 sparsity with 4-bit weight quantization, outperforming prior methods.
  arXiv  Detail & Related papers  (2024-10-12T18:36:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.