Entropy Gate: Entropy Quenching for Near-Lossless Token Compression in LLM Pipelines

• URL: http://arxiv.org/abs/2606.03739v1
• Date: Tue, 02 Jun 2026 14:55:02 GMT
• Title: Entropy Gate: Entropy Quenching for Near-Lossless Token Compression in LLM Pipelines
• Abstract summary: LLM pipelines waste substantial token budgets on low-information content.<n>We introduce Entropy Gate, a token compression framework applying entropy quenching.<n>The framework is stateless, model-agnostic, and deploys as an OpenAI-compatible HTTP proxy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM pipelines waste substantial token budgets on low-information content: repeated context, verbose responses, and redundant boilerplate. We introduce Entropy Gate, a token compression framework applying entropy quenching $-$ a thermodynamic process that progressively freezes out low-energy tokens while preserving semantic fidelity. Each token receives a multi-factor information energy $E(t)$ combining statistical, structural, and positional components. An adaptive quenching schedule $T(τ) = T_0 / (1 + ατ)$ removes tokens whose Boltzmann survival probability $p_i = \exp(-E_i / kT)$ falls below threshold, with a fidelity gate halting compression when energy-weighted similarity drops below $θ$. We prove token selection by descending $E(t)$ maximizes expected semantic preservation, that quenching produces nested survival sets, and that achievable compression approaches the information-theoretic limit $\text{CR} \to 1 - I(P; T)/H(P)$. A Phase 1 heuristic achieves 40-60% compression across five prompt categories while maintaining $S_E > 0.80$, with energy-squared amplification $E \to E^2$ adding 10-25 percentage points. Context deduplication adds 50-70% savings on repeated blocks. Output-side quenching, motivated by findings that brevity improves accuracy, further reduces response overhead. Combined with external memory, reduction composes multiplicatively to 88-96% for agentic workloads. The framework is stateless, model-agnostic, and deploys as an OpenAI-compatible HTTP proxy.

Related papers

• SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models [53.86918766240095]
  Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation.<n>We propose textbfState Anomaly Neutralization (SANE), which applies adaptive $tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure.
  arXiv  Detail & Related papers  (2026-08-23T10:41:07Z)
• ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation [72.54604107217669]
  textscpass@$k$ nearly vanishes after compression, yet textscpass@$k$ recovers substantially under repeated sampling.<n>Recovery should train on the compressed model's own on-policy states with dense token-level supervision.<n>We propose textbfshortopd, a short-to-long OPD schedule that detects teacher-confirmed repetitive suffixes.
  arXiv  Detail & Related papers  (2026-07-14T17:50:50Z)
• Battery-Explicit Energetic Witnesses of CHSH Post-Quantumness [0.0]
  We introduce a trusted-module energetic witness in which a single excitation is conditionally routed into a binary work battery by an energy-preserving SWAP.<n>We show that the classical feed-forward can be embedded into a reversible autonomous module on degenerate logical registers.
  arXiv  Detail & Related papers  (2026-05-09T20:18:44Z)
• Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension [13.891522069967507]
  Spiking transformers achieve competitive accuracy with conventional transformers while offering $38$-$57times$ energy efficiency on neuromorphic hardware.<n>This paper establishes the first comprehensive expressivity theory for spiking self-attention.
  arXiv  Detail & Related papers  (2026-04-17T07:15:53Z)
• Scaling Attention via Feature Sparsity [50.64995497733461]
  Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n2 d)$ cost of self-attention.<n>We propose Sparse Feature Attention, where queries and keys are represented as $k$-sparse codes that preserve high-dimensional expressivity.<n>Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to $2.5times$ and reducing FLOPs and KV-cache by nearly 50%.
  arXiv  Detail & Related papers  (2026-03-17T08:41:50Z)
• A 1/R Law for Kurtosis Contrast in Balanced Mixtures [15.412858526786978]
  Kurtosis-based Independent Component Analysis weakens in wide, balanced mixtures.<n>We prove a sharp redundancy law: for a standardized projection with effective width $R_mathrmeff$, the population excess kurtosis obeys $|(y)|=O(_max/R_mathrmeff)$.<n>We also show that emphpurification -- selecting $m!ll!R$ sign-consistent sources -- restores $R$-independent contrast $ (1/m)$, with a simple data
  arXiv  Detail & Related papers  (2026-02-25T19:01:01Z)
• OneLatent: Single-Token Compression for Visual Latent Reasoning [12.825780556445892]
  We present textbfOneLatent, a framework that compresses intermediate reasoning into a single latent token via supervision from rendered CoT images and DeepSeek-OCR hidden states.<n>Across benchmarks, OneLatent reduces average output length by $11times$ with only a $2.21%$ average accuracy drop relative to textual CoT.<n>On long-chain logical reasoning, OneLatent reaches $99.80%$ on ProntoQA and $97.80%$ on ProsQA with one latent token, with compression up to $87.4times$.
  arXiv  Detail & Related papers  (2026-02-14T12:03:28Z)
• Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space [56.37266873329401]
  Large Language Models (LLMs) apply uniform computation to all tokens, despite language exhibiting highly non-uniform information density.<n>We propose $textbfDynamic Large Concept Models (DLCM)$, a hierarchical language modeling framework that learns semantic boundaries from latent representations and shifts from tokens to a compressed concept space where reasoning is more efficient.
  arXiv  Detail & Related papers  (2025-12-31T04:19:33Z)
• $γ$-FedHT: Stepsize-Aware Hard-Threshold Gradient Compression in Federated Learning [15.458263187587097]
  Gradient compression can effectively alleviate bottlenecks in Federated Learning (FL)<n>We introduce the fundamental conversation of Error-Feedback.<n>We show that $gamma$-FedHT improves accuracy by up to $7.42%$ over Top-$k$ under equal communication.
  arXiv  Detail & Related papers  (2025-05-18T15:55:50Z)
• Fast Rates for Bandit PAC Multiclass Classification [73.17969992976501]
  We study multiclass PAC learning with bandit feedback, where inputs are classified into one of $K$ possible labels and feedback is limited to whether or not the predicted labels are correct. Our main contribution is in designing a novel learning algorithm for the agnostic $(varepsilon,delta)$PAC version of the problem.
  arXiv  Detail & Related papers  (2024-06-18T08:54:04Z)
• Tokenization and the Noiseless Channel [71.25796813073399]
  Good tokenizers lead to emphefficient channel usage, where the channel is the means by which some input is conveyed to the model. In machine translation, we find that across multiple tokenizers, the R'enyi entropy with $alpha = 2.5$ has a very strong correlation with textscBleu: $0.78$ in comparison to just $-0.32$ for compressed length.
  arXiv  Detail & Related papers  (2023-06-29T10:32:09Z)
• Settling the Sample Complexity of Model-Based Offline Reinforcement Learning [50.5790774201146]
  offline reinforcement learning (RL) learns using pre-collected data without further exploration. Prior algorithms or analyses either suffer from suboptimal sample complexities or incur high burn-in cost to reach sample optimality. We demonstrate that the model-based (or "plug-in") approach achieves minimax-optimal sample complexity without burn-in cost.
  arXiv  Detail & Related papers  (2022-04-11T17:26:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.