Information Theory and Statistical Learning

• URL: http://arxiv.org/abs/2605.02989v1
• Date: Mon, 04 May 2026 16:52:14 GMT
• Title: Information Theory and Statistical Learning
• Abstract summary: Learning and information theory intersect in both model training and the characterization of fundamental performance limits.<n>The chapter focuses on the role of divergence measures in model training, with examples ranging from linear and logistic regression to autoregressive models.<n>The treatment of the generative diffusion model provides a more systematic and explicit derivation than is typical in the literature.
• Score: 2.0305676256390934
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This manuscript contains preprint of a chapter under consideration for inclusion in the forthcoming third edition of {\em Cover and Thomas's Elements of Information Theory}, posted with permission from Wiley. The table of contents EIT-3 ToC of the new edition can be found at: https://docs.google.com/document/d/1L-m4oQEJw1PJhoxBeMwrrBD8S_HmvzMEkPbYvS24980/edit?usp=sharing . For feedback, please contact abbas@ee.stanford.edu Learning and information theory intersect in both model training and the characterization of fundamental performance limits. This manuscript provides a concise and accessible treatment of the first intersection, requiring only basic background in information theory and statistics at the senior undergraduate or first-year graduate level. End-of-chapter exercises make the material well suited for classroom use as well as self-study. The chapter focuses on the role of divergence measures in model training, with examples ranging from linear and logistic regression to autoregressive models, variational autoencoders, diffusion models, generative adversarial networks, and score-based models. It introduces the evidence lower bound (ELBO), $f$\!-divergences, and the Fisher divergence. In particular, the treatment of the generative diffusion model provides a more systematic and explicit derivation than is typical in the literature.

Related papers

• Feature-Space Generative Models for One-Shot Class-Incremental Learning [33.77408261771739]
  Few-shot class-incremental learning (FSCIL) is a paradigm where a model, initially trained on a dataset of base classes, must adapt to an expanding problem space by recognizing novel classes with limited data.<n>We propose a novel approach predicated on the hypothesis that base and novel class embeddings have structural similarity.<n>Our approach, Gen1S, consistently improves novel class recognition over the state of the art across multiple benchmarks and backbone architectures.
  arXiv  Detail & Related papers  (2026-01-25T16:45:11Z)
• How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence [52.9442657690445]
  Post-training is essential for the success of large language models (LLMs)<n>We compare base and post-trained LLMs from four perspectives to better understand post-training effects.
  arXiv  Detail & Related papers  (2025-04-03T06:30:55Z)
• Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training [68.94373533768501]
  We model knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training.<n>We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy.
  arXiv  Detail & Related papers  (2025-02-06T13:23:53Z)
• Additive-feature-attribution methods: a review on explainable artificial intelligence for fluid dynamics and heat transfer [1.8434042562191815]
  This review shows that explainability techniques, and in particular additive-feature-attribution methods, are crucial for implementing interpretable deep-learning models in the fluid-mechanics field. The main applications of the additive-feature-attribution methods are introduced, dividing them into three main groups: modeling, fluid-mechanics fundamentals, and applied problems in fluid dynamics and heat transfer turbulence.
  arXiv  Detail & Related papers  (2024-09-18T13:59:02Z)
• The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis [27.310894780313618]
  This paper undertakes a comprehensive comparison of model capabilities at various pretraining intermediate checkpoints. We confirm that specific downstream metrics exhibit similar training dynamics across models of different sizes. In addition to our core findings, we've reproduced Amber and OpenLLaMA, releasing their intermediate checkpoints.
  arXiv  Detail & Related papers  (2024-04-01T16:00:01Z)
• Demystifying Variational Diffusion Models [19.977841588918373]
  Most existing work on diffusion models focuses on either applications or theoretical contributions.<n>We revisit predecessors to diffusion models like hierarchical latent variable models and synthesize a holistic perspective.<n>The resulting narrative is easier to follow as it imposes fewer prerequisites on the average reader.
  arXiv  Detail & Related papers  (2024-01-11T22:37:37Z)
• Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model [74.62272538148245]
  We show that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other. We investigate if it is possible to transfer such "complementary" knowledge from one model to another without performance degradation.
  arXiv  Detail & Related papers  (2023-10-26T17:59:46Z)
• Inducing Causal Structure for Abstractive Text Summarization [76.1000380429553]
  We introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.
  arXiv  Detail & Related papers  (2023-08-24T16:06:36Z)
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models [77.83923746319498]
  We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models. We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
  arXiv  Detail & Related papers  (2023-05-29T04:22:57Z)
• Guided Deep Metric Learning [0.9786690381850356]
  We propose a novel approach to DML that we call Guided Deep Metric Learning. The proposed method is capable of a better manifold generalization and representation to up to 40% improvement.
  arXiv  Detail & Related papers  (2022-06-04T17:34:11Z)
• Algebraic Learning: Towards Interpretable Information Modeling [0.0]
  This thesis addresses the issue of interpretability in general information modeling and endeavors to ease the problem from two scopes. Firstly, a problem-oriented perspective is applied to incorporate knowledge into modeling practice, where interesting mathematical properties emerge naturally. Secondly, given a trained model, various methods could be applied to extract further insights about the underlying system.
  arXiv  Detail & Related papers  (2022-03-13T15:53:39Z)
• Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation [97.22768624862111]
  We analyze NMT models which explicitly evaluates the source and target relative contributions to the generation process. We find that models trained with more data tend to rely on source information more and to have more sharp token contributions.
  arXiv  Detail & Related papers  (2020-10-21T11:37:27Z)
• MED-TEX: Transferring and Explaining Knowledge with Less Data from Pretrained Medical Imaging Models [38.12462659279648]
  A small student model is learned with less data by distilling knowledge from a cumbersome pretrained teacher model. An explainer module is introduced to highlight the regions of an input that are important for the predictions of the teacher model. Our framework outperforms on the knowledge distillation and model interpretation tasks compared to state-of-the-art methods on a fundus dataset.
  arXiv  Detail & Related papers  (2020-08-06T11:50:32Z)
• Explanation-Guided Training for Cross-Domain Few-Shot Classification [96.12873073444091]
  Cross-domain few-shot classification task (CD-FSC) combines few-shot classification with the requirement to generalize across domains represented by datasets. We introduce a novel training approach for existing FSC models. We show that explanation-guided training effectively improves the model generalization.
  arXiv  Detail & Related papers  (2020-07-17T07:28:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.