The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts

• URL: http://arxiv.org/abs/2602.15843v1
• Date: Wed, 21 Jan 2026 18:03:27 GMT
• Title: The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts
• Abstract summary: We find that code generation tolerates aggressive prompt compression (r >= 0.6) while chain-of-thought reasoning degrades gradually.<n>We propose TAAC (Task-Aware Adaptive Compression), achieving 22% cost reduction with 96% quality preservation, outperforming fixed-ratio compression by 7%.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In "Compress or Route?" (Johnson, 2026), we found that code generation tolerates aggressive prompt compression (r >= 0.6) while chain-of-thought reasoning degrades gradually. That study was limited to HumanEval (164 problems), left the "perplexity paradox" mechanism unvalidated, and provided no adaptive algorithm. This paper addresses all three gaps. First, we validate across six code benchmarks (HumanEval, MBPP, HumanEval+, MultiPL-E) and four reasoning benchmarks (GSM8K, MATH, ARC-Challenge, MMLU-STEM), confirming the compression threshold generalizes across languages and difficulties. Second, we conduct the first per-token perplexity analysis (n=723 tokens), revealing a "perplexity paradox": code syntax tokens are preserved (high perplexity) while numerical values in math problems are pruned despite being task-critical (low perplexity). Signature injection recovers +34 percentage points in pass rate (5.3% to 39.3%; Cohen's h=0.890). Third, we propose TAAC (Task-Aware Adaptive Compression), achieving 22% cost reduction with 96% quality preservation, outperforming fixed-ratio compression by 7%. MBPP validation (n=1,800 trials) confirms systematic variation: 3.6% at r=0.3 to 54.6% at r=1.0.

Related papers

• Paritok-4B: Intent-Conditioned Context Compression for Coding Agents [0.9689471757238836]
  Paritok-4B is a 4B LoRA compressor for coding-agent trajectories built on two commitments.<n>It selects spans rather than rewriting them, and 96.4% of the identifiers, paths, and numbers it emits already appear in its input.<n>It is intent-conditioned: told the agent's current task, it acts chiefly inside a retained segment, selecting which lines survive.
  arXiv  Detail & Related papers  (2026-08-25T07:53:30Z)
• Answer First, Reason Later: Commitment Order in Diffusion LLMs [8.902793668615567]
  Masked diffusion language models (dLLMs) can commit tokens in any order.<n>We show that on reasoning tasks this freedom is instead the axis of failure.
  arXiv  Detail & Related papers  (2026-08-06T07:25:04Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers [76.15132587294862]
  Wide-In, Narrow-Out (WINO) is a training-free decoding algorithm that enables revokable parallel generation.<n>WINO+ injects the verified denoising trajectories produced by WINO into model parameters, aligning training with efficient inference.<n>Experiments on LLaDA and MMaDA show that WINO improves both quality and efficiency, while WINO+ further strengthens this progression.
  arXiv  Detail & Related papers  (2026-05-16T11:27:40Z)
• MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents [2.1942030377331245]
  Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets.<n>We introduce MOSAIC-Bench, a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates.<n>We show that nine production coding agents compose innocuous tickets at 53-86% end-to-end ASR with only two refusals across all staged runs.
  arXiv  Detail & Related papers  (2026-05-05T16:38:23Z)
• Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning [0.0]
  We study learned memory tokens as computational scratchpad for a single-block Universal Transformer (UT) on SudokuExtreme.<n>We find that memory tokens are empirically necessary: across all configurations tested, no configuration without memory tokens non-trivial performance.
  arXiv  Detail & Related papers  (2026-04-23T18:30:01Z)
• Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning [0.0]
  We study prompt engineering for formal mathematical reasoning in the context of the SAIR Equational Theories Stage 1 competition.<n>The task requires deciding whether one equational law implies another over all magmas.<n>Over five weeks, we designed, tested, and analyzed more than 40 prompt variants.
  arXiv  Detail & Related papers  (2026-04-20T22:55:23Z)
• Nacrith: Neural Lossless Compression via Ensemble Context Modeling and High-Precision CDF Coding [0.0]
  We present Nacrith, a compression system that achieves the best compression results among the systems evaluated in this study on natural language text.<n>The system requires only 500 MB of GGUF weights and 1.2 GB VRAM per worker, running on consumer GPU.<n>On alice29 (Canterbury Corpus, 152 KB), Nacrith achieves 0.918 bits per byte (bpb), while compressing below the 0th-, 1st-, and 2nd-order Shannon entropy bounds.
  arXiv  Detail & Related papers  (2026-02-23T09:14:05Z)
• ManifoldKV: Training-Free KV Cache Compression via Euclidean Outlier Detection [8.362927764080203]
  We propose a training-free scorer that ranks tokens by Euclidean distance to the key centroid.<n>ManifoldKV achieves 95.7% accuracy at 4K-16K contexts with 20% compression.<n> WindowedManifoldKV restores accuracy to 84.3% at 25% compression, a 49-point recovery over global L2 and +3.2 points over KeyDiff.
  arXiv  Detail & Related papers  (2026-02-09T07:28:55Z)
• DeepPrune: Parallel Scaling without Inter-trace Redundancy [53.62015294143274]
  Over 80% of parallel reasoning traces yield identical final answers, representing substantial wasted computation.<n>We propose DeepPrune, a novel framework that enables efficient parallel scaling through dynamic pruning.<n>Our work establishes a new standard for efficient parallel reasoning, making high-performance reasoning more efficient.
  arXiv  Detail & Related papers  (2025-10-09T17:24:54Z)
• VeriThinker: Learning to Verify Makes Reasoning Model Efficient [52.74493506816969]
  Large Reasoning Models excel at complex tasks using Chain-of-Thought (CoT) reasoning.<n>Their tendency to overthinking leads to unnecessarily lengthy reasoning chains.<n>We introduce VeriThinker, a novel approach for CoT compression.
  arXiv  Detail & Related papers  (2025-05-23T14:17:56Z)
• Not All Tokens Are What You Need In Thinking [34.767739567093656]
  Conditional Token Selection (CTS) identifies and preserves only the most essential tokens in chains of thought.<n>CTS effectively compresses long CoT while maintaining strong reasoning performance.<n>Further reducing training tokens by 42% incurs only a marginal 5% accuracy drop while yielding a 75.8% reduction in reasoning tokens.
  arXiv  Detail & Related papers  (2025-05-23T12:41:29Z)
• Token-by-Token Regeneration and Domain Biases: A Benchmark of LLMs on Advanced Mathematical Problem-Solving [0.0]
  This study evalu-ates 10 large language models (LLMs) with 7 to 8 billion parameters using the MATH dataset.<n>The focus is on their ability to generate executable Python code as a step in their reasoning process, involving over 9,450 code executions.
  arXiv  Detail & Related papers  (2025-01-28T17:11:36Z)
• Preference Optimization for Reasoning with Pseudo Feedback [100.62603571434167]
  We introduce a novel approach to generate pseudo feedback for reasoning tasks by framing the labeling of solutions as an evaluation against associated test cases.<n>We conduct experiments on both mathematical reasoning and coding tasks using pseudo feedback for preference optimization, and observe improvements across both tasks.
  arXiv  Detail & Related papers  (2024-11-25T12:44:02Z)
• Semi-Compressed CRYSTALS-Kyber [4.317605401561789]
  We show it is feasible to reduce the communication overhead of the Kyber by 54%. The improvement is based on two technologies: ciphertext quantization and plaintext encoding. We show that with the Lloyd-Max quantization, 8-PAM, Gray mapping, and a shortened binary BCH(768,638,13) code, the proposed scheme encapsulates 638 bits in a single ciphertext.
  arXiv  Detail & Related papers  (2024-07-25T00:54:22Z)
• Towards Effective and Efficient Non-autoregressive Decoding Using Block-based Attention Mask [74.64216073678617]
  AMD performs parallel NAR inference within contiguous blocks of output labels concealed using attention masks. A beam search algorithm is designed to leverage a dynamic fusion of CTC, AR Decoder, and AMD probabilities. Experiments on the LibriSpeech-100hr corpus suggest the tripartite Decoder incorporating the AMD module produces a maximum decoding speed-up ratio of 1.73x.
  arXiv  Detail & Related papers  (2024-06-14T13:42:38Z)
• Progressive-Hint Prompting Improves Reasoning in Large Language Models [63.98629132836499]
  This paper proposes a new prompting method, named Progressive-Hint Prompting (PHP) It enables automatic multiple interactions between users and Large Language Models (LLMs) by using previously generated answers as hints to progressively guide toward the correct answers. We conducted extensive and comprehensive experiments on seven benchmarks. The results show that PHP significantly improves accuracy while remaining highly efficient.
  arXiv  Detail & Related papers  (2023-04-19T16:29:48Z)
• Faithful Chain-of-Thought Reasoning [51.21714389639417]
  Chain-of-Thought (CoT) prompting boosts Language Models' (LM) performance on a gamut of reasoning tasks. We propose Faithful CoT, a reasoning framework involving two stages: Translation and Problem Solving. This guarantees that the reasoning chain provides a faithful explanation of the final answer.
  arXiv  Detail & Related papers  (2023-01-31T03:04:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.