Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference

• URL: http://arxiv.org/abs/2502.00922v1
• Date: Sun, 02 Feb 2025 21:23:42 GMT
• Title: Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
• Authors: Patrick Yubeaton, Tareq Mahmoud, Shehab Naga, Pooria Taheri, Tianhua Xia, Arun George, Yasmein Khalil, Sai Qian Zhang, Siddharth Joshi, Chinmay Hegde, Siddharth Garg,
• Abstract summary: Large language models (LLMs) have continued to rapidly increase in size.<n>This has exacerbated the difficulty in running state of the art LLMs on small, edge devices.<n>We propose Huff-LLM, a method that lets users store LLM weights in compressed format.
• Score: 19.59857352852377
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As they become more capable, large language models (LLMs) have continued to rapidly increase in size. This has exacerbated the difficulty in running state of the art LLMs on small, edge devices. Standard techniques advocate solving this problem through lossy compression techniques such as quantization or pruning. However, such compression techniques are lossy, and have been shown to change model behavior in unpredictable manners. We propose Huff-LLM, an \emph{end-to-end, lossless} model compression method that lets users store LLM weights in compressed format \emph{everywhere} -- cloud, disk, main memory, and even in on-chip memory/buffers. This allows us to not only load larger models in main memory, but also reduces bandwidth required to load weights on chip, and makes more efficient use of on-chip weight buffers. In addition to the memory savings achieved via compression, we also show latency and energy efficiency improvements when performing inference with the compressed model.

Related papers

• 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float [71.43026659686679]
  Large Language Models (LLMs) have grown rapidly in size, creating challenges for efficient deployment on resource-constrained hardware. We introduce Dynamic-Length Float (DFloat11), a compression framework that reduces LLM size by 30% while preserving outputs that are bit-for-bit identical to the original model.
  arXiv  Detail & Related papers  (2025-04-15T22:38:38Z)
• Embedding Compression Distortion in Video Coding for Machines [67.97469042910855]
  Currently, video transmission serves not only the Human Visual System (HVS) for viewing but also machine perception for analysis. We propose a Compression Distortion Embedding (CDRE) framework, which extracts machine-perception-related distortion representation and embeds it into downstream models. Our framework can effectively boost the rate-task performance of existing codecs with minimal overhead in terms of execution time, and number of parameters.
  arXiv  Detail & Related papers  (2025-03-27T13:01:53Z)
• When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models [12.687035979970194]
  This paper introduces a framework to compress large language models (LLMs) after quantization. A compression-aware quantization is first proposed to enhance model weight compressibility by re-scaling the model parameters before quantization, followed by a pruning method to improve further. Experiments show inference with the compressed model can achieve a 40% reduction in memory size with negligible loss in accuracy and inference speed.
  arXiv  Detail & Related papers  (2025-02-21T13:11:22Z)
• BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments [53.71158537264695]
  Large language models (LLMs) have revolutionized numerous applications, yet their deployment remains challenged by memory constraints on local devices.<n>We introduce textbfBitStack, a novel, training-free weight compression approach that enables megabyte-level trade-offs between memory usage and model performance.
  arXiv  Detail & Related papers  (2024-10-31T13:26:11Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference [1.9639467358416092]
  Transformers have emerged as the backbone of large language models (LLMs) We propose Dynamic Memory Compression (DMC), a method for online key-value cache compression at inference time. We retrofit pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers, achieving up to 7x throughput increase during auto-regressive inference on an NVIDIA H100 GPU.
  arXiv  Detail & Related papers  (2024-03-14T17:59:26Z)
• eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models [19.502740996431452]
  Differentiable KMeans Clustering, or DKM, has shown the state-of-the-art trade-off between compression ratio and accuracy regression. We propose a memory-efficient DKM implementation, eDKM powered by novel techniques to reduce the memory footprint of DKM by orders of magnitudes.
  arXiv  Detail & Related papers  (2023-09-02T15:16:35Z)
• Lossy and Lossless (L$^2$) Post-training Model Size Compression [12.926354646945397]
  We propose a post-training model size compression method that combines lossy and lossless compression in a unified way. Our method can achieve a stable $10times$ compression ratio without sacrificing accuracy and a $20times$ compression ratio with minor accuracy loss in a short time.
  arXiv  Detail & Related papers  (2023-08-08T14:10:16Z)
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression [76.73007709690306]
  We introduce the Sparse-Quantized Representation (SpQR), a new compressed format and quantization technique. SpQR achieves relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs. This makes it possible to run 33B parameter LLM on a single 24 GB consumer GPU without any performance degradation at 15% speedup.
  arXiv  Detail & Related papers  (2023-06-05T17:53:28Z)
• Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt [96.24800696597707]
  We introduce a new perspective to optimize this trade-off by prompting compressed models. We propose a soft prompt learning method where we expose the compressed model to the prompt learning process. Our experimental analysis suggests our soft prompt strategy greatly improves the performance of the 8x compressed LLaMA-7B model.
  arXiv  Detail & Related papers  (2023-05-17T20:45:13Z)
• Single-path Bit Sharing for Automatic Loss-aware Model Compression [126.98903867768732]
  Single-path Bit Sharing (SBS) is able to significantly reduce computational cost while achieving promising performance. Our SBS compressed MobileNetV2 achieves 22.6x Bit-Operation (BOP) reduction with only 0.1% drop in the Top-1 accuracy.
  arXiv  Detail & Related papers  (2021-01-13T08:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.