Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors

• URL: http://arxiv.org/abs/2607.25335v2
• Date: Wed, 29 Jul 2026 12:56:44 GMT
• Title: Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors
• Abstract summary: We conduct evolutionary search over lexical, syntactic, semantic, and discourse seeds to find competitive rule combinations.<n>The resulting linguistic compressor requires no LM forward pass at deployment and uses only CPU-side processing for compression.<n>We evaluate it with a dual-path protocol to balance compression quality and reconstruction fidelity.
• Score: 13.49190771932777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompt compression shortens LLM input to reduce inference cost, yet existing methods score token importance through LM forward passes. It remains questionable whether such nuanced, costly token selection is necessary. Compression requires identifying informative content, a problem that linguistic research has long addressed through cues that can be operationalized as deterministic rules. We therefore ask: can \textbf{linguistic rules alone} serve as effective prompt compressors, without LM-based scoring at compression time? To address this, we conduct offline evolutionary search over lexical, syntactic, semantic, and discourse seeds to find competitive rule combinations. The resulting linguistic compressor requires no LM forward pass at deployment and uses only CPU-side processing for compression. We evaluate it with a dual-path protocol to balance compression quality and reconstruction fidelity. Across short passages, multi-document reasoning, and dialogue-memory QA datasets, evolved compressors achieve performance similar to that of recent advanced prompt-compression strategies. Performance is strongest under light-to-moderate compression and degrades as compression becomes more aggressive, while the Direct and Reconstruction paths exhibit distinct patterns. Evolutionary analysis reveals that effective compression fuses signals across linguistic levels and, as the compression ratio increases, rules shift from token pruning to sentence extraction.

Related papers

• Lossless Prompt Compression via Dictionary-Encoding and In-Context Learning: Enabling Cost-Effective LLM Analysis of Repetitive Data [0.19780197369405136]
  In this paper, we demonstrate that Large Language Models can learn encoding keys in-context and perform analysis on encoded representations.<n>We present a compression algorithm that identifies repetitive patterns at multiple length scales.<n>This training-free approach works with API-based LLMs, directly addressing fundamental deployment constraints.
  arXiv  Detail & Related papers  (2026-03-19T21:05:53Z)
• Cognitive Chunking for Soft Prompts: Accelerating Compressor Learning via Block-wise Causal Masking [28.492055407384495]
  Long contexts increase inference latency, as the computational cost of self-attention grows quadratically with sequence length.<n>Existing methods typically compress the entire context indiscriminately into a set of memory tokens.<n>We propose Parallelized Iterative Compression (PIC), which restricts the receptive field of memory tokens to sequential local chunks.
  arXiv  Detail & Related papers  (2026-02-15T03:58:13Z)
• Detecting Overflow in Compressed Token Representations for Retrieval-Augmented Generation [49.48204107529758]
  We define token overflow as a regime in which compressed representations no longer contain sufficient information to answer a given query.<n>In this paper, we find that query-agnostic saturation statistics reliably separate compressed from uncompressed token representations.<n>Lightweight probing classifiers over both query and context xRAG representations detect overflow with 0.72 AUC-ROC on average.<n>These results advance from query-independent diagnostics to query-aware detectors, enabling low-cost pre-LLM gating to mitigate compression-induced errors.
  arXiv  Detail & Related papers  (2026-02-12T18:15:08Z)
• Arbitrary Ratio Feature Compression via Next Token Prediction [52.10426317889982]
  Arbitrary Ratio Feature Compression (ARFC) framework supports any compression ratio with a single model.<n>ARC is an auto-regressive model that performs compression via next-gressive prediction.<n>MoS module refines the compressed tokens by utilizing multiple compression results.<n>ERGC is integrated into the training process to preserve semantic and structural relationships during compression.
  arXiv  Detail & Related papers  (2026-02-12T02:38:57Z)
• Compressing Many-Shots in In-Context Learning [61.231471139896506]
  We study an approach to improve the memory and computational efficiency of ICL inference by compressing the many-shot prompts.<n>We first show that existing prompt compression methods are ineffective for many-shot compression.<n>We propose MemCom, a layer-wise compression method.
  arXiv  Detail & Related papers  (2025-10-17T16:57:42Z)
• Autoencoding-Free Context Compression for LLMs via Contextual Semantic Anchors [43.02557489472655]
  Current context compression methods rely on autoencoding tasks to train context-agnostic compression tokens to compress contextual semantics.<n>We propose Semantic-Anchor Compression (SAC), a novel method that shifts from autoencoding task based compression to an architecture that is equipped with this compression capability.<n>SAC consistently outperforms existing context compression methods across various compression ratios.
  arXiv  Detail & Related papers  (2025-10-10T01:42:14Z)
• UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression [86.33995240043936]
  UniGist is a sequence-level long-context compression framework for large language models.<n>It efficiently preserves context information by replacing raw tokens with special compression tokens (gists) in a fine-grained manner.<n>Our scheme also supports flexible inference by allowing the actual removal of compressed tokens, resulting in real-time memory savings.
  arXiv  Detail & Related papers  (2025-09-19T08:47:37Z)
• SCOPE: A Generative Approach for LLM Prompt Compression [7.813705327778312]
  We present a novel generative prompt compression method.<n>Unlike the existing token removal methods, our method centers at a chunking-and-summarization mechanism.<n>Our method achieves a significantly better compression quality, and higher stability than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-08-16T01:41:53Z)
• R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search [61.4807238517108]
  Chain-of-Thought (CoT) reasoning enhances large language models (LLMs) by enabling step-by-step problem-solving.<n>CoT's extension to Long-CoT introduces substantial computational overhead due to increased token length.<n>We propose R1-Compress, a two-stage chunk-level compression framework that preserves both local information and coherence.
  arXiv  Detail & Related papers  (2025-05-22T16:06:59Z)
• UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspective [85.08718140718707]
  UNComp is an uncertainty-aware framework that uncovers sparsity patterns that can be used for adaptive compression.<n>By focusing on uncertainty to analyze the sparsity pattern in detail, UNComp reduces the KV cache size to 4.74% of the original, achieves a 6% prefill speedup, and improves throughput by 6.4x.
  arXiv  Detail & Related papers  (2024-10-04T02:32:36Z)
• Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models [21.025001473355996]
  We formalize the problem of prompt compression for large language models (LLMs)<n>We present a framework to unify token-level prompt compression methods which create hard prompts for black-box models.<n>We show that there is a large gap between the performance of current prompt compression methods and the optimal strategy.
  arXiv  Detail & Related papers  (2024-07-22T09:40:13Z)
• In-Context Former: Lightning-fast Compressing Context for Large Language Model [48.831304302467004]
  In this paper, we propose a new approach to compress the long input contexts of Transformer-based large language models (LLMs) We use the cross-attention mechanism and a small number of learnable digest tokens to condense information from the contextual word embeddings. Experimental results indicate that our method requires only 1/32 of the floating-point operations of the baseline during compression and improves processing speed by 68 to 112 times.
  arXiv  Detail & Related papers  (2024-06-19T15:14:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.