OpenZL: A Graph-Based Model for Compression

• URL: http://arxiv.org/abs/2510.03203v2
• Date: Thu, 30 Oct 2025 19:30:14 GMT
• Title: OpenZL: A Graph-Based Model for Compression
• Authors: Yann Collet, Nick Terrell, W. Felix Handte, Danielle Rozenblit, Victor Zhang, Kevin Zhang, Yaelle Goldschlag, Jennifer Lee, Elliot Gorokhovsky, Yonatan Komornik, Daniel Riegel, Stan Angelov, Nadav Rotem,
• Abstract summary: Application-specific compressor systems outperform even the best generic compressors.<n>We show that these challenges can be overcome with a new compression strategy.<n>OpenZL compresses data into a self-describing wire format, any configuration of which can be decompressed by a universal decoder.
• Score: 1.9508265730898475
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Research techniques in the last decade have improved lossless compression ratios by significantly increasing processing time. These techniques have remained obscure because production systems require high throughput and low resource utilization. In practice, application-specific compression algorithms that leverage knowledge of the data structure and semantics are more popular. Application-specific compressor systems outperform even the best generic compressors, but these techniques have some drawbacks. Application-specific compressors are inherently limited in applicability, have high development costs, and are difficult to maintain and deploy. In this work, we show that these challenges can be overcome with a new compression strategy. We propose the "graph model" of compression, a new theoretical framework for representing compression as a directed acyclic graph of modular codecs. OpenZL compresses data into a self-describing wire format, any configuration of which can be decompressed by a universal decoder. OpenZL's design enables rapid development of tailored compressors with minimal code; its universal decoder eliminates deployment lag; and its investment in a well-vetted standard component library minimizes security risks. Experimental results demonstrate that OpenZL achieves superior compression ratios and speeds compared to state-of-the-art general-purpose compressors on a variety of real-world datasets. Internal deployments at Meta have also shown consistent improvements in size and/or speed, with development timelines reduced from months to days. OpenZL thus represents a significant advance in practical, scalable, and maintainable data compression for modern data-intensive applications.

Related papers

• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• Test-Time Steering for Lossless Text Compression via Weighted Product of Experts [27.679089540901007]
  We propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE)<n>At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as that of the best individual model.<n>It seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.
  arXiv  Detail & Related papers  (2025-11-04T16:37:56Z)
• Simple Context Compression: Mean-Pooling and Multi-Ratio Training [12.049015994907629]
  We develop a lightweight and simple mean-pooling approach that consistently outperforms the widely used compression-tokens architecture.<n>We conduct extensive experiments across in-domain and out-of-domain QA datasets, as well as across model families, scales, and compression ratios.<n>Overall, our simple mean-pooling approach achieves the strongest performance, with a relatively small drop when training for multiple compression ratios.
  arXiv  Detail & Related papers  (2025-10-23T17:57:23Z)
• FCA2: Frame Compression-Aware Autoencoder for Modular and Fast Compressed Video Super-Resolution [68.77813885751308]
  State-of-the-art (SOTA) compressed video super-resolution (CVSR) models face persistent challenges, including prolonged inference time, complex training pipelines, and reliance on auxiliary information.<n>We propose an efficient and scalable solution inspired by the structural and statistical similarities between hyperspectral images (HSI) and video data.<n>Our approach introduces a compression-driven dimensionality reduction strategy that reduces computational complexity, accelerates inference, and enhances the extraction of temporal information across frames.
  arXiv  Detail & Related papers  (2025-06-13T07:59:52Z)
• UniPCGC: Towards Practical Point Cloud Geometry Compression via an Efficient Unified Approach [4.754973569457509]
  We propose an efficient unified point cloud geometry compression framework, dubbed as UniPCGC.<n>It supports lossy compression, lossless compression, variable rate and variable complexity.<n>Our method achieves a compression ratio (CR) gain of 8.1% on lossless compression, and a Bjontegaard Delta Rate (BD-Rate) gain of 14.02% on lossy compression.
  arXiv  Detail & Related papers  (2025-03-24T10:51:28Z)
• L3TC: Leveraging RWKV for Learned Lossless Low-Complexity Text Compression [23.179381396167084]
  We introduce a novel Learned Lossless Low-complexity Text Compression method (L3TC)<n> RWKV models achieve the fastest decoding speed with a moderate compression ratio.<n>We propose an outlier-aware tokenizer that uses a limited vocabulary to cover frequent tokens.
  arXiv  Detail & Related papers  (2024-12-21T14:24:32Z)
• Large Language Models for Lossless Image Compression: Next-Pixel Prediction in Language Space is All You Need [53.584140947828004]
  Language large model (LLM) with unprecedented intelligence is a general-purpose lossless compressor for various data modalities. We propose P$2$-LLM, a next-pixel prediction-based LLM, which integrates various elaborated insights and methodologies. Experiments on benchmark datasets demonstrate that P$2$-LLM can beat SOTA classical and learned codecs.
  arXiv  Detail & Related papers  (2024-11-19T12:15:40Z)
• Fast Feedforward 3D Gaussian Splatting Compression [55.149325473447384]
  3D Gaussian Splatting (FCGS) is an optimization-free model that can compress 3DGS representations rapidly in a single feed-forward pass.<n>FCGS achieves a compression ratio of over 20X while maintaining fidelity, surpassing most per-scene SOTA optimization-based methods.
  arXiv  Detail & Related papers  (2024-10-10T15:13:08Z)
• UniCompress: Enhancing Multi-Data Medical Image Compression with Knowledge Distillation [59.3877309501938]
  Implicit Neural Representation (INR) networks have shown remarkable versatility due to their flexible compression ratios. We introduce a codebook containing frequency domain information as a prior input to the INR network. This enhances the representational power of INR and provides distinctive conditioning for different image blocks.
  arXiv  Detail & Related papers  (2024-05-27T05:52:13Z)
• Deep Lossy Plus Residual Coding for Lossless and Near-lossless Image Compression [85.93207826513192]
  We propose a unified and powerful deep lossy plus residual (DLPR) coding framework for both lossless and near-lossless image compression. We solve the joint lossy and residual compression problem in the approach of VAEs. In the near-lossless mode, we quantize the original residuals to satisfy a given $ell_infty$ error bound.
  arXiv  Detail & Related papers  (2022-09-11T12:11:56Z)
• A Fast Transformer-based General-Purpose Lossless Compressor [19.5544227045828]
  We introduce transformer into deep learning compressors to build history-dependencies in parallel. Existing transformer is too heavy in computation and incompatible to compression tasks. Byte-grouping and Shared-ffn schemes are proposed to fully utilize the capacity of the single-layer transformer.
  arXiv  Detail & Related papers  (2022-03-30T07:46:19Z)
• Exploring Autoencoder-based Error-bounded Compression for Scientific Data [14.724393511470225]
  We develop an error-bounded autoencoder-based framework in terms of the SZ model. We optimize the compression quality for the main stages in our designed AE-based error-bounded compression framework.
  arXiv  Detail & Related papers  (2021-05-25T07:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.