Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction

• URL: http://arxiv.org/abs/2605.29000v1
• Date: Wed, 27 May 2026 18:58:30 GMT
• Title: Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction
• Abstract summary: We study emphlossy semantic text compression, where the encoder strategically deletes parts of the text.<n>We benchmark a progression of deletion strategies, including uniform step deletion, word-length-guided deletion (WordLen), word-frequency-guided deletion (WordFreq), LP-optimized deletion (Opt), entropy-based deletion using GPT-2 surprisal, and hybrid methods that combine frequency and surprisal signals.
• Score: 3.008906408145323
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional lossless text compression preserves every byte, but its gains on natural language are often modest in realistic operating regimes. We study \emph{lossy semantic text compression}, where the encoder strategically deletes parts of the text and a large language model (LLM) reconstructs the original content from the retained skeleton. We benchmark a progression of deletion strategies, including uniform step deletion, word-length-guided deletion (WordLen), word-frequency-guided deletion (WordFreq), LP-optimized deletion (Opt), entropy-based deletion using GPT-2 surprisal, and hybrid methods that combine frequency and surprisal signals. Evaluation on the BBC News dataset across retention rates $\r_{keep} \in [0.1,0.9]$ shows three main findings. First, WordFreq is a strong low-cost baseline: despite using only a static frequency lookup, it remains competitive with much more expensive semantic methods while being far faster at the encoder. Second, semantic and hybrid methods provide their clearest gains at mild-to-moderate compression, whereas word-frequency deletion is often more robust at the lowest retention rates. Third, QLoRA fine-tuning yields a strong local decoder that is competitive with Gemini 2.0 Flash and is often strongest in decoder-only comparisons. Additional English and Chinese experiments show that the overall framework transfers across domains, while the best deletion rule remains dataset-dependent.

Related papers

• VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference [53.883871460875234]
  VoxZip is a semantic-anchored KV cache compression framework.<n>First stage uses automatic speech recognition (ASR) transcriptions as explicit semantic anchors to temporally align, compress, and fuse audio tokens.<n>Second stage employs a dynamic filtering strategy based on temporally decayed accumulated attention to evict non-essential tokens.
  arXiv  Detail & Related papers  (2026-08-09T08:17:18Z)
• Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression? [0.0]
  We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs.<n>A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace.<n>These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression.
  arXiv  Detail & Related papers  (2026-07-19T22:19:17Z)
• STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation [60.597357847359255]
  Video segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries.<n>Existing methods address this through token compression, yet typically operate on features lacking temporal context.<n>State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $mathcalO(T)$ cost compression to reduce.
  arXiv  Detail & Related papers  (2026-07-03T03:28:25Z)
• Beyond Clean Text: Evaluating Encoder and Decoder Robustness for Bangla Event Detection in Noisy Text [0.0]
  Event detection systems are typically evaluated on clean, curated text, leaving their robustness to real-world noise largely unexplored.<n>We introduce a generalized Bangla news event and a benchmark comprising 9,979 annotated sentences across 40 event subtypes.<n>We show that encoder models achieve higher performance on clean text but degrade substantially under noise, whereas decoder-only LLMs are markedly more robust.
  arXiv  Detail & Related papers  (2026-06-29T21:03:32Z)
• CANDLE: Character-level Arabic Noise Deduplication using Lightweight Encoder [1.0399316033225514]
  We present CANDLE, a lightweight system for character-level Arabic noise deduplication.<n>At the heart of CANDLE is a novel application of Connectionist Temporal Classification (CTC) to this task.<n>The CTC model achieves a Sentence Error Rate (SER) as low as $5.37%$ and consistently outperforms a classification-based baseline by a large margin.
  arXiv  Detail & Related papers  (2026-06-23T16:17:24Z)
• SemanticZip: A Pilot Framework for Lossy Text Compression with LLMs as Semantic Decompressors [0.0]
  We study a more aggressive but explicitly lossy setting: compress text into compact codes that an LLM can expand into task-relevant meaning.<n>Unlike ordinary summarization, SemanticZip does not require byte-identical reconstruction.<n>This paper is a pilot framework, not a benchmark claim.
  arXiv  Detail & Related papers  (2026-05-23T12:14:04Z)
• Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data [0.19780197369405136]
  In this paper, we demonstrate that Large Language Models can learn encoding keys in-context and perform analysis on encoded representations.<n>We present a compression algorithm that identifies repetitive patterns at multiple length scales.<n>This training-free approach works with API-based LLMs, directly addressing fundamental deployment constraints.
  arXiv  Detail & Related papers  (2026-03-19T21:05:53Z)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression [55.63153956934198]
  Chain-of-Thought (CoT) reasoning successfully enhances the reasoning capabilities of Large Language Models (LLMs)<n>Existing CoT compression methods often suffer from a critical loss of logical fidelity at high compression ratios.<n>We propose a novel EXTreme-RAtio Chain-of-Thought Compression framework, termed Extra-CoT, which aggressively reduces the token budget while preserving answer accuracy.
  arXiv  Detail & Related papers  (2026-02-09T06:57:15Z)
• SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec [83.61175662066364]
  Speech codecs serve as a crucial bridge in unifying speech and text language models.<n>Existing methods face several challenges in semantic encoding.<n>We propose SecoustiCodec, a cross-modal aligned low-bitrate streaming speech codecs.
  arXiv  Detail & Related papers  (2025-08-04T19:22:14Z)
• Memory-Efficient FastText: A Comprehensive Approach Using Double-Array Trie Structures and Mark-Compact Memory Management [0.0]
  FastText has established itself as a fundamental algorithm for learning word representations.<n>However, its hash-based bucketing mechanism introduces critical limitations for large-scale industrial deployment.<n>This paper presents a memory optimization framework that reimagines FastText's memory management.
  arXiv  Detail & Related papers  (2025-06-02T02:11:22Z)
• ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inference [61.412894960600205]
  Large Language Models (LLMs) require significant GPU memory when processing long texts.<n>ChunkKV reimagines KV cache compression by treating semantic chunks as basic compression units.<n>Result: ChunkKV outperforms state-of-the-art methods by up to 8.7% in precision.
  arXiv  Detail & Related papers  (2025-02-01T03:49:47Z)
• Training LLMs over Neurally Compressed Text [55.11828645767342]
  This paper explores the idea of training large language models (LLMs) over highly compressed text.<n>We propose Equal-Info Windows, a novel compression technique whereby text is segmented into blocks that each compress to the same bit length.<n>We demonstrate effective learning over neurally compressed text that improves with scale, and outperforms byte-level baselines by a wide margin on perplexity and inference speed benchmarks.
  arXiv  Detail & Related papers  (2024-04-04T17:48:28Z)
• Semantic Compression With Large Language Models [1.0874100424278175]
  Large language models (LLMs) are revolutionizing information retrieval, question answering, summarization, and code generation tasks. LLMs are inherently limited by the number of input and output tokens that can be processed at once. This paper presents three contributions to research on LLMs.
  arXiv  Detail & Related papers  (2023-04-25T01:47:05Z)
• Syntactically Look-Ahead Attention Network for Sentence Compression [36.6256383447417]
  Sentence compression is the task of compressing a long sentence into a short one by deleting redundant words. In sequence-to-sequence (Seq2Seq) based models, the decoder unidirectionally decides to retain or delete words. We propose a novel Seq2Seq model, syntactically look-ahead attention network (SLAHAN), that can generate informative summaries.
  arXiv  Detail & Related papers  (2020-02-04T06:26:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.