Rate-Distortion Optimization for Transformer Inference

• URL: http://arxiv.org/abs/2601.22002v1
• Date: Thu, 29 Jan 2026 17:12:46 GMT
• Title: Rate-Distortion Optimization for Transformer Inference
• Authors: Anderson de Andrade, Alon Harell, Ivan V. Bajić,
• Abstract summary: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference.<n>We introduce a principled rate-distortion-based framework for lossy compression that learns compact encodings that explicitly trade off against accuracy.
• Score: 1.5378391391800512
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Transformers achieve superior performance on many tasks, but impose heavy compute and memory requirements during inference. This inference can be made more efficient by partitioning the process across multiple devices, which, in turn, requires compressing its intermediate representations. In this work, we introduce a principled rate-distortion-based framework for lossy compression that learns compact encodings that explicitly trade off bitrate against accuracy. Experiments on language benchmarks show that the proposed codec achieves substantial savings with improved accuracy in some cases, outperforming more complex baseline methods. We characterize and analyze the rate-distortion performance of transformers, offering a unified lens for understanding performance in representation coding. This formulation extends information-theoretic concepts to define the gap between rate and entropy, and derive some of its bounds. We further develop probably approximately correct (PAC)-style bounds for estimating this gap. For different architectures and tasks, we empirically demonstrate that their rates are driven by these bounds, adding to the explainability of the formulation.

Related papers

• Accelerate Speculative Decoding with Sparse Computation in Verification [49.74839681322316]
  Speculative decoding accelerates autoregressive language model inference by verifying multiple draft tokens in parallel.<n>Existing sparsification methods are designed primarily for standard token-by-token autoregressive decoding.<n>We propose a sparse verification framework that jointly sparsifies attention, FFN, and MoE components during the verification stage to reduce the dominant computation cost.
  arXiv  Detail & Related papers  (2025-12-26T07:53:41Z)
• Rethinking Autoregressive Models for Lossless Image Compression via Hierarchical Parallelism and Progressive Adaptation [75.58269386927076]
  Autoregressive (AR) models are often dismissed as impractical due to prohibitive computational cost.<n>This work re-thinks this paradigm, introducing a framework built on hierarchical parallelism and progressive adaptation.<n> Experiments on diverse datasets (natural, satellite, medical) validate that our method achieves new state-of-the-art compression.
  arXiv  Detail & Related papers  (2025-11-14T06:27:58Z)
• Fast Data-independent KLT Approximations Based on Integer Functions [0.0]
  The Karhunen-Loeve transform (KLT) stands as a well-established discrete transform, demonstrating optimal characteristics in data decorrelation and dimensionality reduction. This paper introduces a category of low-complexity, data-independent KLT approximations, employing a range of round-off functions. The proposed transforms perform well when compared to the exact KLT and approximations considering classical performance measures.
  arXiv  Detail & Related papers  (2024-10-11T20:05:05Z)
• Accelerating Error Correction Code Transformers [56.75773430667148]
  We introduce a novel acceleration method for transformer-based decoders. We achieve a 90% compression ratio and reduce arithmetic operation energy consumption by at least 224 times on modern hardware.
  arXiv  Detail & Related papers  (2024-10-08T11:07:55Z)
• Corner-to-Center Long-range Context Model for Efficient Learned Image Compression [70.0411436929495]
  In the framework of learned image compression, the context model plays a pivotal role in capturing the dependencies among latent representations. We propose the textbfCorner-to-Center transformer-based Context Model (C$3$M) designed to enhance context and latent predictions. In addition, to enlarge the receptive field in the analysis and synthesis transformation, we use the Long-range Crossing Attention Module (LCAM) in the encoder/decoder.
  arXiv  Detail & Related papers  (2023-11-29T21:40:28Z)
• Compression with Bayesian Implicit Neural Representations [16.593537431810237]
  We propose overfitting variational neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding instead of quantizing and entropy coding it. Experiments show that our method achieves strong performance on image and audio compression while retaining simplicity.
  arXiv  Detail & Related papers  (2023-05-30T16:29:52Z)
• HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression [69.36555801766762]
  We propose a hardware-aware tensor decomposition framework, dubbed HEAT, that enables efficient exploration of the exponential space of possible decompositions. We experimentally show that our hardware-aware factorized BERT variants reduce the energy-delay product by 5.7x with less than 1.1% accuracy loss.
  arXiv  Detail & Related papers  (2022-11-30T05:31:45Z)
• Learned Image Compression with Generalized Octave Convolution and Cross-Resolution Parameter Estimation [5.238765582868391]
  We propose a learned multi-resolution image compression framework, which exploits octave convolutions to factorize the latent representations into the high-resolution (HR) and low-resolution (LR) parts. Experimental results show that our method separately reduces the decoding time by approximately 73.35 % and 93.44 % compared with that of state-of-the-art learned image compression methods.
  arXiv  Detail & Related papers  (2022-09-07T08:21:52Z)
• Learning Quantization in LDPC Decoders [14.37550972719183]
  We propose a floating-point surrogate model that imitates quantization effects as additions of uniform noise. A deep learning-based method is then applied to optimize the message bitwidths. We report an error-rate performance within 0.2 dB of floating-point decoding at an average message quantization bitwidth of 3.1 bits.
  arXiv  Detail & Related papers  (2022-08-10T07:07:54Z)
• Unified Multivariate Gaussian Mixture for Efficient Neural Image Compression [151.3826781154146]
  latent variables with priors and hyperpriors is an essential problem in variational image compression. We find inter-correlations and intra-correlations exist when observing latent variables in a vectorized perspective. Our model has better rate-distortion performance and an impressive $3.18times$ compression speed up.
  arXiv  Detail & Related papers  (2022-03-21T11:44:17Z)
• The Rate-Distortion-Accuracy Tradeoff: JPEG Case Study [30.84385779593074]
  We focus on the design of the quantization tables in the JPEG compression standard. We offer a novel optimal tuning of these tables via continuous optimization. We report a substantial boost in performance by a simple and easily implemented modification of these tables.
  arXiv  Detail & Related papers  (2020-08-03T01:39:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.