Progressive Cramming: Reliable Token Compression and What It Reveals

• URL: http://arxiv.org/abs/2607.21231v1
• Date: Thu, 23 Jul 2026 11:46:36 GMT
• Title: Progressive Cramming: Reliable Token Compression and What It Reveals
• Abstract summary: Token cramming compresses sequences into learned embeddings with near-perfect reconstruction.<n>We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable.
• Score: 5.7789316833104225
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99\% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding's interactions in the model's early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.

Related papers

• Complete Suffix Prediction for Recommendation via Latent Retrieval over Process Graphs [0.06524460254566904]
  We propose a graphbased metric-learning framework that reformulates complete suffix prediction as latent retrieval over process graphs.<n>We show that graph-based latent retrieval is an effective alternative to sequential suffix prediction for recommendation-oriented process monitoring.
  arXiv  Detail & Related papers  (2026-09-14T14:59:08Z)
• Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space [78.05046281165885]
  Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form.<n>GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes.
  arXiv  Detail & Related papers  (2026-08-31T14:54:30Z)
• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression? [0.0]
  We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs.<n>A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace.<n>These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression.
  arXiv  Detail & Related papers  (2026-07-19T22:19:17Z)
• ChunkFlow: Towards Continuity-Consistent Chunked Policy Learning [5.727608625370017]
  ChunkFlow is a seam-aware training-and-execution framework for chunked policies.<n>It partitions each chunk into frozen, editable, and future zones, applies deterministic overlap blending at execution, and trains raw predictions with seam and first- and second-order continuity losses.<n> Experiments on CALVIN, LIBERO, and real robots show an improved success-stability trade-off with low-latency inference.
  arXiv  Detail & Related papers  (2026-07-14T17:43:25Z)
• Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models [0.0]
  We study the hidden-state inversion problem as a continuous embedding-space optimisation.<n>We analyse which tokens break the reconstructions and find a sharp categorical asymmetry.<n>Last-layer hidden states of GPT-2 are as sensitive as original text.
  arXiv  Detail & Related papers  (2026-07-01T12:18:12Z)
• Next-Acceleration-Scale Prediction for Autoregressive MRI Reconstruction [52.32112533846212]
  MRI reconstruction is an inherently ill-posed inverse problem, since incomplete measurements admit many plausible solutions.<n>We address this limitation by moving reconstruction to discrete multi-scale latent space and posing it as autoregressive next-acceleration-scale prediction.<n>We show that our approach delivers improved reconstruction performance across diverse sampling patterns under extreme undersampling.
  arXiv  Detail & Related papers  (2026-05-19T04:40:50Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• TokenTrim: Inference-Time Token Pruning for Autoregressive Long Video Generation [45.36298679288268]
  Auto-regressive video generation enables long video synthesis by iteratively conditioning each new batch of frames on previously generated content.<n>Recent work has shown that such pipelines suffer from severe temporal drift, where errors accumulate and amplify over long horizons.<n>We propose a simple, inference-time method that mitigates temporal drift by identifying and removing unstable latent tokens before they are reused for conditioning.
  arXiv  Detail & Related papers  (2026-01-30T19:44:16Z)
• OmniSAT: Compact Action Token, Faster Auto Regression [70.70037017501357]
  We introduce an Omni Swift Action Tokenizer, which learns a compact, transferable action representation.<n>The resulting discrete tokenization shortens the training sequence by 6.8$times$, and lowers the target entropy.
  arXiv  Detail & Related papers  (2025-10-08T03:55:24Z)
• Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking [50.465604300990904]
  Grokking refers to the abrupt improvement in test accuracy after extended overfitting.<n>In this work, we investigate the grokking mechanism underlying the Transformer in the task of prime number operations.
  arXiv  Detail & Related papers  (2025-04-04T04:42:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.