Learning is Forgetting: LLM Training As Lossy Compression

• URL: http://arxiv.org/abs/2604.07569v1
• Date: Wed, 08 Apr 2026 20:12:07 GMT
• Title: Learning is Forgetting: LLM Training As Lossy Compression
• Abstract summary: We argue large language models (LLMs) are best seen as an instance of lossy compression, where over training they learn by retaining only information relevant to their objective(s)<n>We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression.
• Score: 19.24753945415483
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the increasing prevalence of large language models (LLMs), we still have a limited understanding of how their representational spaces are structured. This limits our ability to interpret how and what they learn or relate them to learning in humans. We argue LLMs are best seen as an instance of lossy compression, where over training they learn by retaining only information in their training data relevant to their objective(s). We show pre-training results in models that are optimally compressed for next-sequence prediction, approaching the Information Bottleneck bound on compression. Across an array of open weights models, each compresses differently, likely due to differences in the data and training recipes used. However even across different families of LLMs the optimality of a model's compression, and the information present in it, can predict downstream performance on across a wide array of benchmarks, letting us directly link representational structure to actionable insights about model performance. In the general case the work presented here offers a unified Information-Theoretic framing for how these models learn that is deployable at scale.

Related papers

• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Compression Laws for Large Language Models [20.62274005080048]
  We introduce compression laws for language language models (LLMs)<n>We empirically examine the effects of structured model compression on LLMs through over $1000$ experiments.<n>Our findings indicate that the test cross-entropy loss increases quadratically with the compression ratio.
  arXiv  Detail & Related papers  (2025-04-06T03:39:34Z)
• How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence [52.9442657690445]
  Post-training is essential for the success of large language models (LLMs)<n>We compare base and post-trained LLMs from four perspectives to better understand post-training effects.
  arXiv  Detail & Related papers  (2025-04-03T06:30:55Z)
• Disentangling Fine-Tuning from Pre-Training in Visual Captioning with Hybrid Markov Logic [2.113770213797994]
  We learn a probabilistic model using Hybrid Markov Logic Networks (HMLNs) over the training examples.<n>For a generated caption, we quantify the influence of training examples based on the HMLN distribution.
  arXiv  Detail & Related papers  (2025-03-18T02:39:26Z)
• Slamming: Training a Speech Language Model on One GPU in a Day [20.802090523583196]
  Slam is a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours.<n>We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost.
  arXiv  Detail & Related papers  (2025-02-19T17:21:15Z)
• Measuring Sample Importance in Data Pruning for Language Models based on Information Entropy [4.079147243688765]
  We consider a data pruning method based on information entropy.<n>We propose that the samples in the training corpus be ranked in terms of their informativeness.<n> Experiments reveal that the proposed information-based pruning can improve upon various language modeling and downstream tasks.
  arXiv  Detail & Related papers  (2024-06-20T09:09:34Z)
• Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression [64.07696663255155]
  Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, the massive size of these models poses huge challenges for their deployment in real-world applications. We introduce a novel compression paradigm called Retrieval-based Knowledge Transfer (RetriKT) which effectively transfers the knowledge of LLMs to extremely small-scale models.
  arXiv  Detail & Related papers  (2023-10-24T07:58:20Z)
• Language models are weak learners [71.33837923104808]
  We show that prompt-based large language models can operate effectively as weak learners. We incorporate these models into a boosting approach, which can leverage the knowledge within the model to outperform traditional tree-based boosting. Results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
  arXiv  Detail & Related papers  (2023-06-25T02:39:19Z)
• CodeGen2: Lessons for Training LLMs on Programming and Natural Languages [116.74407069443895]
  We unify encoder and decoder-based models into a single prefix-LM. For learning methods, we explore the claim of a "free lunch" hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored.
  arXiv  Detail & Related papers  (2023-05-03T17:55:25Z)
• Few-Shot Non-Parametric Learning with Deep Latent Variable Model [50.746273235463754]
  We propose Non-Parametric learning by Compression with Latent Variables (NPC-LV) NPC-LV is a learning framework for any dataset with abundant unlabeled data but very few labeled ones. We show that NPC-LV outperforms supervised methods on all three datasets on image classification in low data regime.
  arXiv  Detail & Related papers  (2022-06-23T09:35:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.