Understanding is Compression

• URL: http://arxiv.org/abs/2407.07723v2
• Date: Wed, 21 Aug 2024 02:45:36 GMT
• Title: Understanding is Compression
• Authors: Ziguang Li, Chao Huang, Xuliang Wang, Haibo Hu, Cole Wyeth, Dongbo Bu, Quan Yu, Wen Gao, Xingwu Liu, Ming Li,
• Abstract summary: 6G communication speed requirement raises open question for revolutionary new ideas of data compression. We show large language models (LLMs) understand data better than ever before. We present LMCompress, which shatters all previous compression algorithms.
• Score: 18.747845226548456
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Modern data compression methods are slowly reaching their limits after 80 years of research, millions of papers, and wide range of applications. Yet, the extravagant 6G communication speed requirement raises a major open question for revolutionary new ideas of data compression. We have previously shown all understanding or learning are compression, under reasonable assumptions. Large language models (LLMs) understand data better than ever before. Can they help us to compress data? The LLMs may be seen to approximate the uncomputable Solomonoff induction. Therefore, under this new uncomputable paradigm, we present LMCompress. LMCompress shatters all previous lossless compression algorithms, doubling the lossless compression ratios of JPEG-XL for images, FLAC for audios, and H.264 for videos, and quadrupling the compression ratio of bz2 for texts. The better a large model understands the data, the better LMCompress compresses.

Related papers

• FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression [1.9699843876565526]
  FineZip is a novel text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance.
  arXiv  Detail & Related papers  (2024-09-25T17:58:35Z)
• AlphaZip: Neural Network-Enhanced Lossless Text Compression [0.0]
  This paper introduces a lossless text compression approach using a Large Language Model (LLM) The method involves two key steps: first, prediction using a dense neural network architecture, such as a transformer block; second, compressing the predicted ranks with standard compression algorithms like Adaptive Huffman, LZ77, or Gzip.
  arXiv  Detail & Related papers  (2024-09-23T14:21:06Z)
• 500xCompressor: Generalized Prompt Compression for Large Language Models [32.4489985319054]
  500xCompressor is a method that compresses extensive natural language contexts into a minimum of one single special token. It is designed to compress any text, answer various types of questions, and could be utilized by the original large language model (LLM) without requiring fine-tuning.
  arXiv  Detail & Related papers  (2024-08-06T10:51:47Z)
• LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression [43.048684907893104]
  This paper focuses on task-agnostic prompt compression for better generalizability and efficiency. We formulate prompt compression as a token classification problem to guarantee the faithfulness of the compressed prompt to the original one. Our approach leads to lower latency by explicitly learning the compression objective with smaller models such as XLM-RoBERTa-large and mBERT.
  arXiv  Detail & Related papers  (2024-03-19T17:59:56Z)
• MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model [78.4051835615796]
  This paper proposes a method called Multimodal Image Semantic Compression. It consists of an LMM encoder for extracting the semantic information of the image, a map encoder to locate the region corresponding to the semantic, an image encoder generates an extremely compressed bitstream, and a decoder reconstructs the image based on the above information. It can achieve optimal consistency and perception results while saving perceptual 50%, which has strong potential applications in the next generation of storage and communication.
  arXiv  Detail & Related papers  (2024-02-26T17:11:11Z)
• Lossy and Lossless (L$^2$) Post-training Model Size Compression [12.926354646945397]
  We propose a post-training model size compression method that combines lossy and lossless compression in a unified way. Our method can achieve a stable $10times$ compression ratio without sacrificing accuracy and a $20times$ compression ratio with minor accuracy loss in a short time.
  arXiv  Detail & Related papers  (2023-08-08T14:10:16Z)
• Deep Lossy Plus Residual Coding for Lossless and Near-lossless Image Compression [85.93207826513192]
  We propose a unified and powerful deep lossy plus residual (DLPR) coding framework for both lossless and near-lossless image compression. We solve the joint lossy and residual compression problem in the approach of VAEs. In the near-lossless mode, we quantize the original residuals to satisfy a given $ell_infty$ error bound.
  arXiv  Detail & Related papers  (2022-09-11T12:11:56Z)
• Cross Modal Compression: Towards Human-comprehensible Semantic Compression [73.89616626853913]
  Cross modal compression is a semantic compression framework for visual data. We show that our proposed CMC can achieve encouraging reconstructed results with an ultrahigh compression ratio.
  arXiv  Detail & Related papers  (2022-09-06T15:31:11Z)
• PILC: Practical Image Lossless Compression with an End-to-end GPU Oriented Neural Framework [88.18310777246735]
  We propose an end-to-end image compression framework that achieves 200 MB/s for both compression and decompression with a single NVIDIA Tesla V100 GPU. Experiments show that our framework compresses better than PNG by a margin of 30% in multiple datasets.
  arXiv  Detail & Related papers  (2022-06-10T03:00:10Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)
• Towards Robust Data Hiding Against (JPEG) Compression: A Pseudo-Differentiable Deep Learning Approach [78.05383266222285]
  It is still an open challenge to achieve the goal of data hiding that can be against these compressions. Deep learning has shown large success in data hiding, while non-differentiability of JPEG makes it challenging to train a deep pipeline for improving robustness against lossy compression. In this work, we propose a simple yet effective approach to address all the above limitations at once.
  arXiv  Detail & Related papers  (2020-12-30T12:30:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.