TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding

• URL: http://arxiv.org/abs/2606.08184v1
• Date: Sat, 06 Jun 2026 14:12:54 GMT
• Title: TextEconomizer: Enhancing Lossy Text Compression with Denoising Transformers and Entropy Coding
• Abstract summary: Lossy text compression reduces data size while preserving core meaning, making it well-suited for summarization, automated analysis, and digital archives.<n>Despite the dominance of transformer-based models in language modeling, integrating context vectors and entropy coding into Sequence-to-Sequence generation remains underexplored.<n>We introduce TextEconomizer, an encoder-decoder framework paired with a transformer neural network that reduces variable-sized inputs by 50% to 80% without prior knowledge of dataset dimensions.
• Score: 0.8644909837301148
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lossy text compression reduces data size while preserving core meaning, making it well-suited for summarization, automated analysis, and digital archives. Despite the dominance of transformer-based models in language modeling, integrating context vectors and entropy coding into Sequence-to-Sequence (Seq2Seq) generation remains underexplored. A key challenge lies in identifying the most informative context vectors from encoder output and incorporating entropy coding to enhance storage efficiency while maintaining high-quality outputs, even under noisy text. We introduce TextEconomizer, an encoder-decoder framework paired with a transformer neural network that reduces variable-sized inputs by 50% to 80% without prior knowledge of dataset dimensions. Our model achieves competitive compression ratios via entropy coding while delivering near-perfect text quality, assessed by BLEU, ROUGE, METEOR, and semantic similarity scores. TextEconomizer operates with approximately 153x fewer parameters than comparable models, achieving a 5.39x compression ratio without sacrificing semantic quality. We also evaluate an LSTM-based autoencoder achieving a state-of-the-art 67x compression ratio with 196x fewer parameters, and LLaMAFormer, a modified transformer with 263x fewer parameters than ICAE while maintaining competitive text quality. TextEconomizer significantly surpasses existing transformer-based models in balancing memory efficiency and high-fidelity outputs, marking a breakthrough in lossy compression with optimal space utilization.

Related papers

• Transformers from Compressed Representations [74.48571451824569]
  TEMPEST (TransformErs froM comPressed rEpreSenTations) is a method that exploits the inherent byte-stream structure of compressed files to design an effective tokenization and encoding strategy.<n>Our proposal substantially reduces the number of tokens required for semantic classification, thereby lowering both computational complexity and memory usage.
  arXiv  Detail & Related papers  (2025-10-26T13:48:03Z)
• REGEN: Learning Compact Video Embedding with (Re-)Generative Decoder [52.698595889988766]
  We present a novel perspective on learning video embedders for generative modeling.<n>Rather than requiring an exact reproduction of an input video, an effective embedder should focus on visually plausible reconstructions.<n>We propose replacing the conventional encoder-decoder video embedder with an encoder-generator framework.
  arXiv  Detail & Related papers  (2025-03-11T17:51:07Z)
• An Enhanced Text Compression Approach Using Transformer-based Language Models [1.2937020918620652]
  We propose a transformer-based method named RejuvenateForme for text decompression.<n>Our meticulous pre-processing technique incorporates the Le-Ziv-Welch algorithm.<n>The RejuvenateForme achieves a BLEU score of 27.31, 25.78, and 50.45 on the EN-DE, EN-FR, and BookCorpus corpora, showcasing its comprehensive efficacy.
  arXiv  Detail & Related papers  (2024-12-15T03:01:17Z)
• Accelerating Error Correction Code Transformers [56.75773430667148]
  We introduce a novel acceleration method for transformer-based decoders. We achieve a 90% compression ratio and reduce arithmetic operation energy consumption by at least 224 times on modern hardware.
  arXiv  Detail & Related papers  (2024-10-08T11:07:55Z)
• Epsilon-VAE: Denoising as Visual Decoding [61.29255979767292]
  We propose denoising as decoding, shifting from single-step reconstruction to iterative refinement.<n>Specifically, we replace the decoder with a diffusion process that iteratively refines noise to recover the original image.<n>By adopting iterative reconstruction through diffusion, our autoencoder, namely Epsilon-VAE, achieves high reconstruction quality.
  arXiv  Detail & Related papers  (2024-10-05T08:27:53Z)
• ConvNeXt-ChARM: ConvNeXt-based Transform for Efficient Neural Image Compression [18.05997169440533]
  We propose ConvNeXt-ChARM, an efficient ConvNeXt-based transform coding framework, paired with a compute-efficient channel-wise auto-regressive auto-regressive. We show that ConvNeXt-ChARM brings consistent and significant BD-rate (PSNR) reductions estimated on average to 5.24% and 1.22% over the versatile video coding (VVC) reference encoder (VTM-18.0) and the state-of-the-art learned image compression method SwinT-ChARM.
  arXiv  Detail & Related papers  (2023-07-12T11:45:54Z)
• Compressing Pre-trained Transformers via Low-Bit NxM Sparsity for Natural Language Understanding [20.75335227098455]
  Large pre-trained Transformer networks have demonstrated dramatic improvements in many natural language understanding tasks. New hardware supporting both NM semi-structured sparsity and low-precision integer computation is a promising solution to boost model serving efficiency. We propose a flexible compression framework NxMiFormer that performs simultaneous sparsification and quantization.
  arXiv  Detail & Related papers  (2022-06-30T04:33:50Z)
• Text Compression-aided Transformer Encoding [77.16960983003271]
  We propose explicit and implicit text compression approaches to enhance the Transformer encoding. backbone information, meaning the gist of the input text, is not specifically focused on. Our evaluation on benchmark datasets shows that the proposed explicit and implicit text compression approaches improve results in comparison to strong baselines.
  arXiv  Detail & Related papers  (2021-02-11T11:28:39Z)
• Hamming OCR: A Locality Sensitive Hashing Neural Network for Scene Text Recognition [14.250874536962366]
  Self-attention-based scene text recognition approaches have achieved outstanding performance. Number of parameters in both classification and embedding layers is independent on the size of vocabulary. Hamming OCR achieves competitive results.
  arXiv  Detail & Related papers  (2020-09-23T01:20:19Z)
• Modeling Lost Information in Lossy Image Compression [72.69327382643549]
  Lossy image compression is one of the most commonly used operators for digital images. We propose a novel invertible framework called Invertible Lossy Compression (ILC) to largely mitigate the information loss problem.
  arXiv  Detail & Related papers  (2020-06-22T04:04:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.