Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

• URL: http://arxiv.org/abs/2607.11883v1
• Date: Mon, 13 Jul 2026 17:58:50 GMT
• Title: Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
• Abstract summary: We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution.<n>The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart.
• Score: 55.04392122319273
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.

Related papers

• Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space [78.05046281165885]
  Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form.<n>GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes.
  arXiv  Detail & Related papers  (2026-08-31T14:54:30Z)
• Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios [76.85739138203014]
  We present SpecFormer, a novel architecture that accelerates unidirectional and attention mechanisms.<n>We demonstrate that SpecFormer achieves lower training demands and reduced computational costs.
  arXiv  Detail & Related papers  (2025-11-25T14:20:08Z)
• LLMComp: A Language Modeling Paradigm for Error-Bounded Scientific Data Compression (Technical Report) [4.2414540423650795]
  LLMCOMP is a lossy compression paradigm that leverages decoder-only large language models to model scientific data.<n>It consistently outperforms state-of-the-art compressors, achieving up to 30% higher compression ratios under strict error bounds.
  arXiv  Detail & Related papers  (2025-10-24T05:41:04Z)
• Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression [57.54335545892155]
  We introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook.<n>Our approach achieves a better trade-off between model size and accuracy compared to existing post-training quantization baselines.
  arXiv  Detail & Related papers  (2025-10-23T20:19:48Z)
• UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance [65.01483640267885]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet code generation remains a major challenge.<n>We introduce UnitCoder, a systematic pipeline leveraging model-generated unit tests to guide and validate the code generation process.<n>Our work presents a scalable approach that leverages model-generated unit tests to guide the synthesis of high-quality code data from pre-training corpora.
  arXiv  Detail & Related papers  (2025-02-17T05:37:02Z)
• Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough? [11.865113785648932]
  We explore the extent to which code-comment coherence, a specific quality attribute of code summaries, can be used to optimize code summarization datasets.<n>We examine multiple levels of training instances from two state-of-the-art datasets (TL-CodeSum and Funcom) and evaluate the resulting models on three manually curated test sets.
  arXiv  Detail & Related papers  (2025-02-11T15:02:19Z)
• The Persian Rug: solving toy models of superposition using large-scale symmetries [0.0]
  We present a complete mechanistic description of the algorithm learned by a minimal non-linear sparse data autoencoder in the limit of large input dimension. Our work contributes to neural network interpretability by introducing techniques for understanding the structure of autoencoders.
  arXiv  Detail & Related papers  (2024-10-15T22:52:45Z)
• Few-Shot Non-Parametric Learning with Deep Latent Variable Model [50.746273235463754]
  We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV) NPC-LV is a learning framework for any dataset with abundant unlabeled data but very few labeled ones. We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime.
  arXiv  Detail & Related papers  (2022-06-23T09:35:03Z)
• COIN++: Data Agnostic Neural Compression [55.27113889737545]
  COIN++ is a neural compression framework that seamlessly handles a wide range of data modalities. We demonstrate the effectiveness of our method by compressing various data modalities.
  arXiv  Detail & Related papers  (2022-01-30T20:12:04Z)
• Generative time series models using Neural ODE in Variational Autoencoders [0.0]
  We implement Neural Ordinary Differential Equations in a Variational Autoencoder setting for generative time series modeling. An object-oriented approach to the code was taken to allow for easier development and research.
  arXiv  Detail & Related papers  (2022-01-12T14:38:11Z)
• A flexible, extensible software framework for model compression based on the LC algorithm [10.787390511207683]
  We propose a software framework that allows a user to compress a neural network or other machine learning model with minimal effort. The library is written in Python and PyTorch and available in Github.
  arXiv  Detail & Related papers  (2020-05-15T21:14:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.