Efficient Learned Image Compression Through Knowledge Distillation

• URL: http://arxiv.org/abs/2509.10366v1
• Date: Fri, 12 Sep 2025 15:59:55 GMT
• Title: Efficient Learned Image Compression Through Knowledge Distillation
• Authors: Fabien Allemand, Attilio Fiandrotti, Sumanta Chaudhuri, Alaa Eddine Mazouz,
• Abstract summary: Learned image compression sits at the intersection of machine learning and image processing.<n>Recent research suggests that neural network-based compression models consistently outperform conventional codecs.<n>This study aims to reduce the resource requirements of neural networks used for image compression by leveraging knowledge distillation.
• Score: 6.5170521662453185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learned image compression sits at the intersection of machine learning and image processing. With advances in deep learning, neural network-based compression methods have emerged. In this process, an encoder maps the image to a low-dimensional latent space, which is then quantized, entropy-coded into a binary bitstream, and transmitted to the receiver. At the receiver end, the bitstream is entropy-decoded, and a decoder reconstructs an approximation of the original image. Recent research suggests that these models consistently outperform conventional codecs. However, they require significant processing power, making them unsuitable for real-time use on resource-constrained platforms, which hinders their deployment in mainstream applications. This study aims to reduce the resource requirements of neural networks used for image compression by leveraging knowledge distillation, a training paradigm where smaller neural networks, partially trained on the outputs of larger, more complex models, can achieve better performance than when trained independently. Our work demonstrates that knowledge distillation can be effectively applied to image compression tasks: i) across various architecture sizes, ii) to achieve different image quality/bit rate tradeoffs, and iii) to save processing and energy resources. This approach introduces new settings and hyperparameters, and future research could explore the impact of different teacher models, as well as alternative loss functions. Knowledge distillation could also be extended to transformer-based models. The code is publicly available at: https://github.com/FABallemand/PRIM .

Related papers

• Compressing image encoders via latent distillation [14.979912631427334]
  Deep learning models for image compression often face practical limitations in hardware-constrained applications.<n>We propose a methodology to partially compress these networks by reducing the size of their encoders.<n>Our approach uses a simplified knowledge distillation strategy to approximate the latent space of the original models with less data and shorter training, yielding lightweight encoders from heavyweight ones.
  arXiv  Detail & Related papers  (2026-01-09T08:50:38Z)
• Multi-Scale Invertible Neural Network for Wide-Range Variable-Rate Learned Image Compression [90.59962443790593]
  In this paper, we present a variable-rate image compression model based on invertible transform to overcome limitations.<n> Specifically, we design a lightweight multi-scale invertible neural network, which maps the input image into multi-scale latent representations.<n> Experimental results demonstrate that the proposed method achieves state-of-the-art performance compared to existing variable-rate methods.
  arXiv  Detail & Related papers  (2025-03-27T09:08:39Z)
• Streaming Lossless Volumetric Compression of Medical Images Using Gated Recurrent Convolutional Neural Network [0.0]
  This paper introduces a hardware-friendly streaming lossless volumetric compression framework. We propose a gated recurrent convolutional neural network that combines diverse convolutional structures and fusion gate mechanisms. Our method exhibits robust generalization ability and competitive compression speed.
  arXiv  Detail & Related papers  (2023-11-27T07:19:09Z)
• Lossy Image Compression with Quantized Hierarchical VAEs [33.173021636656465]
  ResNet VAEs are originally designed for data (image) distribution modeling. We present a powerful and efficient model that outperforms previous methods on natural image lossy compression. Our model compresses images in a coarse-to-fine fashion and supports parallel encoding and decoding.
  arXiv  Detail & Related papers  (2022-08-27T17:15:38Z)
• Convolutional Neural Network (CNN) to reduce construction loss in JPEG compression caused by Discrete Fourier Transform (DFT) [0.0]
  Convolutional Neural Networks (CNN) have received more attention than most other types of deep neural networks. In this work, an effective image compression method is purposed using autoencoders.
  arXiv  Detail & Related papers  (2022-08-26T12:46:16Z)
• The Devil Is in the Details: Window-based Attention for Image Compression [58.1577742463617]
  Most existing learned image compression models are based on Convolutional Neural Networks (CNNs) In this paper, we study the effects of multiple kinds of attention mechanisms for local features learning, then introduce a more straightforward yet effective window-based local attention block. The proposed window-based attention is very flexible which could work as a plug-and-play component to enhance CNN and Transformer models.
  arXiv  Detail & Related papers  (2022-03-16T07:55:49Z)
• Implicit Neural Representations for Image Compression [103.78615661013623]
  Implicit Neural Representations (INRs) have gained attention as a novel and effective representation for various data types. We propose the first comprehensive compression pipeline based on INRs including quantization, quantization-aware retraining and entropy coding. We find that our approach to source compression with INRs vastly outperforms similar prior work.
  arXiv  Detail & Related papers  (2021-12-08T13:02:53Z)
• Joint Global and Local Hierarchical Priors for Learned Image Compression [30.44884350320053]
  Recently, learned image compression methods have shown superior performance compared to the traditional hand-crafted image codecs. We propose a novel entropy model called Information Transformer (Informer) that exploits both local and global information in a content-dependent manner. Our experiments demonstrate that Informer improves rate-distortion performance over the state-of-the-art methods on the Kodak and Tecnick datasets.
  arXiv  Detail & Related papers  (2021-12-08T06:17:37Z)
• Restormer: Efficient Transformer for High-Resolution Image Restoration [118.9617735769827]
  convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data. Transformers have shown significant performance gains on natural language and high-level vision tasks. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks.
  arXiv  Detail & Related papers  (2021-11-18T18:59:10Z)
• Supervised Compression for Resource-constrained Edge Computing Systems [26.676557573171618]
  Full-scale deep neural networks are often too resource-intensive in terms of energy and storage. This paper adopts ideas from knowledge distillation and neural image compression to compress intermediate feature representations more efficiently. It achieves better supervised rate-distortion performance while also maintaining smaller end-to-end latency.
  arXiv  Detail & Related papers  (2021-08-21T11:10:29Z)
• Modeling Lost Information in Lossy Image Compression [72.69327382643549]
  Lossy image compression is one of the most commonly used operators for digital images. We propose a novel invertible framework called Invertible Lossy Compression (ILC) to largely mitigate the information loss problem.
  arXiv  Detail & Related papers  (2020-06-22T04:04:56Z)
• Learning End-to-End Lossy Image Compression: A Benchmark [90.35363142246806]
  We first conduct a comprehensive literature survey of learned image compression methods. We describe milestones in cutting-edge learned image-compression methods, review a broad range of existing works, and provide insights into their historical development routes. By introducing a coarse-to-fine hyperprior model for entropy estimation and signal reconstruction, we achieve improved rate-distortion performance.
  arXiv  Detail & Related papers  (2020-02-10T13:13:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.