ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

• URL: http://arxiv.org/abs/2603.17435v1
• Date: Wed, 18 Mar 2026 07:21:21 GMT
• Title: ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression
• Abstract summary: Lossless model compression holds tremendous promise for alleviating the memory and bandwidth bottlenecks in bit-exact Large Language Model (LLM) serving.<n>Existing approaches often result in substantial inference slowdowns due to fundamental design mismatches with GPU architectures.<n>We present ZipServ, a lossless compression framework co-designed for efficient LLM inference.
• Score: 19.538318240352424
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lossless model compression holds tremendous promise for alleviating the memory and bandwidth bottlenecks in bit-exact Large Language Model (LLM) serving. However, existing approaches often result in substantial inference slowdowns due to fundamental design mismatches with GPU architectures: at the kernel level, variable-length bitstreams produced by traditional entropy codecs break SIMT parallelism; at the system level, decoupled pipelines lead to redundant memory traffic. We present ZipServ, a lossless compression framework co-designed for efficient LLM inference. ZipServ introduces Tensor-Core-Aware Triple Bitmap Encoding (TCA-TBE), a novel fixed-length format that enables constant-time, parallel decoding, together with a fused decompression-GEMM (ZipGEMM) kernel that decompresses weights on-the-fly directly into Tensor Core registers. This "load-compressed, compute-decompressed" design eliminates intermediate buffers and maximizes compute intensity. Experiments show that ZipServ reduces the model size by up to 30%, achieves up to 2.21x kernel-level speedup over NVIDIA's cuBLAS, and expedites end-to-end inference by an average of 1.22x over vLLM. ZipServ is the first lossless compression system that provides both storage savings and substantial acceleration for LLM inference on GPUs.

Related papers

• SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving [4.543073806867322]
  SplitZip is a GPU-friendly compressor for KV cache transfer.<n>SplitZip exploits redundancy in floating-point exponents of KV activations.<n>An offline top-16 exponent codebook eliminates online-histogramming.
  arXiv  Detail & Related papers  (2026-05-03T04:22:51Z)
• UCCL-Zip: Lossless Compression Supercharged GPU Communication [37.95759514379356]
  UCCL-Zip supports both point-to-point (P2P) and collective communication without modifying user-facing APIs or compromising numerical correctness.<n>Uzip-P2P employs a split-send pipeline that exposes transmissible data early and overlaps compression with communication.<n>Uzip-NCCL integrates compression into NCCL's persistent kernel model via fused execution.
  arXiv  Detail & Related papers  (2026-04-19T00:05:36Z)
• Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference [5.608398371429037]
  LLMLingua achieves up to 18% end-to-end speed-ups when prompt length, compression ratio, and hardware capacity are well matched.<n>We show that effective compression can reduce memory usage enough to offload workloads from data center GPUs to commodity cards, with only a 0.3s increase in latency.
  arXiv  Detail & Related papers  (2026-04-03T11:41:53Z)
• DeltaKV: Residual-Based KV Cache Compression via Long-Range Similarity [50.52392445266824]
  We propose a residual-based KV cache compression framework motivated by long-range inter-token similarity and highly shared latent components in KV representations.<n>Instead of discarding tokens, DeltaKV encodes semantic residuals relative to retrieved historical references, preserving fidelity while substantially reducing storage.<n>Experiments show that DeltaKV reduces KV cache memory to 29% of the original while maintaining near-lossless accuracy on LongBench, SCBench, and AIME.
  arXiv  Detail & Related papers  (2026-02-08T15:14:36Z)
• Spava: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention [63.69228529380251]
  Spava is a sequence-parallel framework with optimized attention for long-video inference.<n>Spava delivers speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss.
  arXiv  Detail & Related papers  (2026-01-29T09:23:13Z)
• CXL-SpecKV: A Disaggregated FPGA Speculative KV-Cache for Datacenter LLM Serving [5.216774377033164]
  Large Language Models (LLMs) have revolutionized natural language processing tasks.<n>LLMs face challenges due to the massive memory requirements of key-value ( KV) caches.<n>We propose textbfCXL-SpecKV, a novel disaggregated KV-cache architecture.
  arXiv  Detail & Related papers  (2025-12-11T15:40:36Z)
• SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression [7.603859408568262]
  Large Language Models (LLMs) face a significant bottleneck during autoregressive inference due to the massive memory footprint of the Key-Value (KV) cache.<n>We introduce SWAN, a novel, fine-tuning-free framework that eliminates this overhead.<n>Our method uses an offline matrix to rotate and prune the KV-cache, which is then used directly in the attention computation without any reconstruction.
  arXiv  Detail & Related papers  (2025-11-24T09:41:24Z)
• FlashSVD: Memory-Efficient Inference with Streaming for Low-Rank Models [15.244129138320782]
  FlashSVD is an end-to-end rank-aware streaming inference framework for SVD-compressed large language models.<n>It cuts peak activation memory by up to 70.2% and intermediate transient memory by 75%.<n>It incurs no accuracy loss with upstream encodering compression methods, offering a practical path toward memory-constrained deployment of low-rank LLMs.
  arXiv  Detail & Related papers  (2025-08-02T22:06:46Z)
• DistZO2: High-Throughput and Memory-Efficient Zeroth-Order Fine-tuning LLMs with Distributed Parallel Computing [4.589472292598182]
  Fine-tuning large language models (LLMs) remains resource-intensive due to their sheer scale.<n>We present DistZO2, a memory-efficient framework for distributed zeroth-order fine-tuning of LLMs.
  arXiv  Detail & Related papers  (2025-07-03T22:53:34Z)
• Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference [19.59857352852377]
  Large language models (LLMs) have continued to rapidly increase in size.<n>This has exacerbated the difficulty in running state of the art LLMs on small, edge devices.<n>We propose Huff-LLM, a method that lets users store LLM weights in compressed format.
  arXiv  Detail & Related papers  (2025-02-02T21:23:42Z)
• LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy [59.1298692559785]
  Key-Value ( KV) cache is crucial component in serving transformer-based autoregressive large language models (LLMs) Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages; (2) KV cache compression at test time; and (3) KV cache compression at test time. We propose a low-rank approximation of KV weight matrices, allowing plug-in integration with existing transformer-based LLMs without model retraining. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages.
  arXiv  Detail & Related papers  (2024-10-04T03:10:53Z)
• MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models [58.3342517278868]
  This paper describes the design of Mixed-precision AutoRegressive LINear kernels. It shows that batchsizes up to 16-32 can be supported with close to maximum ($4times$) quantization speedup. MarLIN accomplishes this via a combination of techniques, such as asynchronous memory access, complex task scheduling and pipelining.
  arXiv  Detail & Related papers  (2024-08-21T16:10:41Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• ZipLM: Inference-Aware Structured Pruning of Language Models [56.52030193434863]
  We propose a novel structured compression approach for large language models (LLMs) called ZipLM. ZipLM achieves state-of-the-art accuracy-vs-speedup, while matching a set of desired target runtime speedups. ZipLM produces state-of-the-art compressed models across all settings.
  arXiv  Detail & Related papers  (2023-02-07T18:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.