Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs

• URL: http://arxiv.org/abs/2408.08575v1
• Date: Fri, 16 Aug 2024 07:23:18 GMT
• Title: Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs
• Authors: Jinming Liu, Yuntao Wei, Junyan Lin, Shengyang Zhao, Heming Sun, Zhibo Chen, Wenjun Zeng, Xin Jin,
• Abstract summary: We present a new image compression paradigm to achieve intelligently coding for machine'' by cleverly leveraging the common sense of Large Multimodal Models (LMMs) We dub our method textitSDComp'' for textitSemantically textitDisentangled textitCompression'', and compare it with state-of-the-art codecs on a wide variety of different vision tasks.
• Score: 47.7670923159071
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a new image compression paradigm to achieve ``intelligently coding for machine'' by cleverly leveraging the common sense of Large Multimodal Models (LMMs). We are motivated by the evidence that large language/multimodal models are powerful general-purpose semantics predictors for understanding the real world. Different from traditional image compression typically optimized for human eyes, the image coding for machines (ICM) framework we focus on requires the compressed bitstream to more comply with different downstream intelligent analysis tasks. To this end, we employ LMM to \textcolor{red}{tell codec what to compress}: 1) first utilize the powerful semantic understanding capability of LMMs w.r.t object grounding, identification, and importance ranking via prompts, to disentangle image content before compression, 2) and then based on these semantic priors we accordingly encode and transmit objects of the image in order with a structured bitstream. In this way, diverse vision benchmarks including image classification, object detection, instance segmentation, etc., can be well supported with such a semantically structured bitstream. We dub our method ``\textit{SDComp}'' for ``\textit{S}emantically \textit{D}isentangled \textit{Comp}ression'', and compare it with state-of-the-art codecs on a wide variety of different vision tasks. SDComp codec leads to more flexible reconstruction results, promised decoded visual quality, and a more generic/satisfactory intelligent task-supporting ability.

Related papers

• Exploiting Inter-Image Similarity Prior for Low-Bitrate Remote Sensing Image Compression [10.427300958330816]
  We propose a codebook-based RS image compression (Code-RSIC) method with a generated discrete codebook. The code significantly outperforms state-of-the-art traditional and learning-based image compression algorithms in terms of perception quality.
  arXiv  Detail & Related papers  (2024-07-17T03:33:16Z)
• CMC-Bench: Towards a New Paradigm of Visual Signal Compression [85.1839779884282]
  We introduce CMC-Bench, a benchmark of the cooperative performance of Image-to-Text (I2T) and Text-to-Image (T2I) models for image compression. At ultra-lows, this paper proves that the combination of some I2T and T2I models has surpassed the most advanced visual signal protocols.
  arXiv  Detail & Related papers  (2024-06-13T17:41:37Z)
• SMC++: Masked Learning of Unsupervised Video Semantic Compression [54.62883091552163]
  We propose a Masked Video Modeling (MVM)-powered compression framework that particularly preserves video semantics. MVM is proficient at learning generalizable semantics through the masked patch prediction task. It may also encode non-semantic information like trivial textural details, wasting bitcost and bringing semantic noises.
  arXiv  Detail & Related papers  (2024-06-07T09:06:40Z)
• MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Model [78.4051835615796]
  This paper proposes a method called Multimodal Image Semantic Compression. It consists of an LMM encoder for extracting the semantic information of the image, a map encoder to locate the region corresponding to the semantic, an image encoder generates an extremely compressed bitstream, and a decoder reconstructs the image based on the above information. It can achieve optimal consistency and perception results while saving perceptual 50%, which has strong potential applications in the next generation of storage and communication.
  arXiv  Detail & Related papers  (2024-02-26T17:11:11Z)
• Preprocessing Enhanced Image Compression for Machine Vision [14.895698385236937]
  We propose a preprocessing enhanced image compression method for machine vision tasks. Our framework is built upon the traditional non-differential codecs. Experimental results show our method achieves a better tradeoff between the coding and the performance of the downstream machine vision tasks by saving about 20%.
  arXiv  Detail & Related papers  (2022-06-12T03:36:38Z)
• A New Image Codec Paradigm for Human and Machine Uses [53.48873918537017]
  A new scalable image paradigm for both human and machine uses is proposed in this work. The high-level instance segmentation map and the low-level signal features are extracted with neural networks. An image is designed and trained to achieve the general-quality image reconstruction with the 16-bit gray-scale profile and signal features.
  arXiv  Detail & Related papers  (2021-12-19T06:17:38Z)
• How to Exploit the Transferability of Learned Image Compression to Conventional Codecs [25.622863999901874]
  We show how learned image coding can be used as a surrogate to optimize an image for encoding. Our approach can remodel a conventional image to adjust for the MS-SSIM distortion with over 20% rate improvement without any decoding overhead.
  arXiv  Detail & Related papers  (2020-12-03T12:34:51Z)
• Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics [127.65410486227007]
  Video coding, which targets to compress and reconstruct the whole frame, and feature compression, which only preserves and transmits the most critical information, stand at two ends of the scale. Recent endeavors in imminent trends of video compression, e.g. deep learning based coding tools and end-to-end image/video coding, and MPEG-7 compact feature descriptor standards, promote the sustainable and fast development in their own directions. In this paper, thanks to booming AI technology, e.g. prediction and generation models, we carry out exploration in the new area, Video Coding for Machines (VCM), arising from the emerging MPEG
  arXiv  Detail & Related papers  (2020-01-10T17:24:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.