Minimum Description Length of a Spectrum Variational Autoencoder: A Theory

• URL: http://arxiv.org/abs/2504.00395v1
• Date: Tue, 01 Apr 2025 03:37:18 GMT
• Title: Minimum Description Length of a Spectrum Variational Autoencoder: A Theory
• Abstract summary: We argue that Minimum Description Length (MDL) is essential to deep learning.<n>We introduce a novel theoretical framework for designing and evaluating deep Variational Autoencoders (VAEs) based on MDL.<n>We claim that a Spectrum VAE understands the data distribution in the most appropriate way when the MDL is achieved.
• Score: 1.3735277588793995
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) trained through end-to-end learning have achieved remarkable success across diverse machine learning tasks, yet they are not explicitly designed to adhere to the Minimum Description Length (MDL) principle, which posits that the best model provides the shortest description of the data. In this paper, we argue that MDL is essential to deep learning and propose a further generalized principle: Understanding is the use of a small amount of information to represent a large amount of information. To this end, we introduce a novel theoretical framework for designing and evaluating deep Variational Autoencoders (VAEs) based on MDL. In our theory, we designed the Spectrum VAE, a specific VAE architecture whose MDL can be rigorously evaluated under given conditions. Additionally, we introduce the concept of latent dimension combination, or pattern of spectrum, and provide the first theoretical analysis of their role in achieving MDL. We claim that a Spectrum VAE understands the data distribution in the most appropriate way when the MDL is achieved. This work is entirely theoretical and lays the foundation for future research on designing deep learning systems that explicitly adhere to information-theoretic principles.

Related papers

• There Will Be a Scientific Theory of Deep Learning [62.690977616250954]
  We make the case that a scientific theory of deep learning is emerging.<n>We identify five growing bodies of work that point toward such a theory.<n>We argue that the emerging theory is best thought of as a mechanics of the learning process.
  arXiv  Detail & Related papers  (2026-04-23T13:58:12Z)
• Understanding Degradation with Vision Language Model [56.09241449206817]
  Understanding visual degradations is a critical yet challenging problem in computer vision.<n>We introduce DU-VLM, a multimodal chain-of-thought model trained with supervised fine-tuning and reinforcement learning.<n>We also introduce textbfDU-110k, a large-scale dataset comprising 110,000 clean-degraded pairs with grounded physical annotations.
  arXiv  Detail & Related papers  (2026-02-04T13:51:15Z)
• Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimality [52.57416398859353]
  We show that causal minimality can endow latent representations of diffusion vision and autoregressive language models with clear causal interpretation and robust, component-wise identifiable control.<n>We introduce a novel theoretical framework for hierarchical selection models, where higher-level concepts emerge from the constrained composition of lower-level variables.<n>These causally grounded concepts serve as levers for fine-grained model steering, paving the way for transparent, reliable systems.
  arXiv  Detail & Related papers  (2025-12-11T14:59:14Z)
• On the Theoretical Foundation of Sparse Dictionary Learning in Mechanistic Interpretability [5.009082958329585]
  We develop the first unified theoretical framework considering sparse dictionary learning (SDL) as one unified optimization problem.<n>We provide the first theoretical explanations for some empirically observed phenomena, including feature absorption, dead neurons, and the neuron resampling technique.
  arXiv  Detail & Related papers  (2025-12-05T08:47:19Z)
• How LLMs Learn to Reason: A Complex Network Perspective [14.638878448692493]
  Training large language models with Reinforcement Learning from Verifiable Rewards exhibits a set of puzzling behaviors.<n>We propose that these seemingly disparate phenomena can be explained using a single unifying theory.<n>Our work provides a new physical intuition for engineering the emergent reasoning capabilities of future AI systems.
  arXiv  Detail & Related papers  (2025-09-28T04:10:37Z)
• Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition [86.21199607040147]
  Self-Improving cognition (SIcog) is a self-learning framework for constructing next-generation foundation language models.<n>We introduce Chain-of-Description, a step-by-step visual understanding method, and integrate structured chain-of-thought (CoT) reasoning to support in-depth multimodal reasoning.<n>Extensive experiments demonstrate that SIcog produces next-generation foundation MLLMs with substantially improved multimodal cognition.
  arXiv  Detail & Related papers  (2025-03-16T00:25:13Z)
• I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? [76.15163242945813]
  Large language models (LLMs) have led many to conclude that they exhibit a form of intelligence.<n>We introduce a novel generative model that generates tokens on the basis of human-interpretable concepts represented as latent discrete variables.
  arXiv  Detail & Related papers  (2025-03-12T01:21:17Z)
• Learning Mask Invariant Mutual Information for Masked Image Modeling [35.63719638508299]
  Maskedencodes (MAEs) represent a prominent self-supervised learning paradigm in computer vision. Recent studies have attempted to elucidate the functioning of MAEs through contrastive learning and feature representation analysis. We propose a new perspective for understanding MAEs by leveraging the information bottleneck principle in information theory.
  arXiv  Detail & Related papers  (2025-02-27T03:19:05Z)
• Coding for Intelligence from the Perspective of Category [66.14012258680992]
  Coding targets compressing and reconstructing data, and intelligence. Recent trends demonstrate the potential homogeneity of these two fields. We propose a novel problem of Coding for Intelligence from the category theory view.
  arXiv  Detail & Related papers  (2024-07-01T07:05:44Z)
• Probability Distribution Learning and Its Application in Deep Learning [0.0]
  This paper introduces a novel theoretical learning framework, termed probability distribution learning (PD learning)<n>PD learning focuses on learning the underlying probability distribution, which is modeled as a random variable within the probability simplex.
  arXiv  Detail & Related papers  (2024-06-09T06:49:22Z)
• Understanding Encoder-Decoder Structures in Machine Learning Using Information Measures [10.066310107046084]
  We present new results to model and understand the role of encoder-decoder design in machine learning (ML) We use two main information concepts, information sufficiency (IS) and mutual information loss (MIL), to represent predictive structures in machine learning.
  arXiv  Detail & Related papers  (2024-05-30T19:58:01Z)
• Towards Optimal Learning of Language Models [124.65669486710992]
  We present a theory for the optimal learning of language models (LMs) We derive a theorem, named Learning Law, to reveal the properties of the dynamics in the optimal learning process under our objective. We empirically verify that the optimal learning of LMs essentially stems from the improvement of the coefficients in the scaling law of LMs.
  arXiv  Detail & Related papers  (2024-02-27T18:52:19Z)
• Towards an Information Theoretic Framework of Context-Based Offline Meta-Reinforcement Learning [48.79569442193824]
  We show that COMRL algorithms are essentially optimizing the same mutual information objective between the task variable $M$ and its latent representation $Z$ by implementing various approximate bounds.<n>As demonstrations, we propose a supervised and a self-supervised implementation of $I(Z; M)$, and empirically show that the corresponding optimization algorithms exhibit remarkable generalization across a broad spectrum of RL benchmarks.<n>This work lays the information theoretic foundation for COMRL methods, leading to a better understanding of task representation learning in the context of reinforcement learning.
  arXiv  Detail & Related papers  (2024-02-04T09:58:42Z)
• Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention [53.896974148579346]
  Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. The enigmatic black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. We propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs.
  arXiv  Detail & Related papers  (2023-12-22T19:55:58Z)
• Evaluating and Explaining Large Language Models for Code Using Syntactic Structures [74.93762031957883]
  This paper introduces ASTxplainer, an explainability method specific to Large Language Models for code. At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes. We perform an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects.
  arXiv  Detail & Related papers  (2023-08-07T18:50:57Z)
• Explaining Explainability: Towards Deeper Actionable Insights into Deep Learning through Second-order Explainability [70.60433013657693]
  Second-order explainable AI (SOXAI) was recently proposed to extend explainable AI (XAI) from the instance level to the dataset level. We demonstrate for the first time, via example classification and segmentation cases, that eliminating irrelevant concepts from the training set based on actionable insights from SOXAI can enhance a model's performance.
  arXiv  Detail & Related papers  (2023-06-14T23:24:01Z)
• Latent Variable Models in the Era of Industrial Big Data: Extension and Beyond [7.361977372607915]
  latent variable models (LVMs) and their counterparts account for a major share and play a vital role in many industrial modeling areas. We propose a novel concept called lightweight deep LVM (LDLVM)
  arXiv  Detail & Related papers  (2022-08-23T09:58:37Z)
• Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics [30.365274034429508]
  We argue that a future deep learning theory should inherit three characteristics: a textitarchhierically structured network architecture, parameters textititeratively optimized using gradient-based methods, and information from the data that evolves textitcompressively We integrate these characteristics into a graphical model called textitneurashed, which effectively explains some common empirical patterns in deep learning.
  arXiv  Detail & Related papers  (2021-12-17T19:51:26Z)
• Error Bounds for a Matrix-Vector Product Approximation with Deep ReLU Neural Networks [0.0]
  Theory of deep learning has spurred the theory of deep learning-oriented depth and breadth of developments. Motivated by such developments, we pose fundamental questions: can we accurately approximate an arbitrary matrix-vector product using deep rectified linear unit (ReLU) feedforward neural networks (FNNs)? We derive error bounds in Lebesgue and Sobolev norms that comprise our developed deep approximation theory. The developed theory is also applicable for guiding and easing the training of teacher deep ReLU FNNs in view of the emerging teacher-student AI or ML paradigms.
  arXiv  Detail & Related papers  (2021-11-25T08:14:55Z)
• An Empirical Investigation into Deep and Shallow Rule Learning [0.0]
  In this paper, we empirically compare deep and shallow rule learning with a uniform general algorithm. Our experiments on both artificial and real-world benchmark data indicate that deep rule networks outperform shallow networks.
  arXiv  Detail & Related papers  (2021-06-18T17:43:17Z)
• Understanding Interpretability by generalized distillation in Supervised Classification [3.5473853445215897]
  Recent interpretation strategies focus on human understanding of the underlying decision mechanisms of the complex Machine Learning models. We propose an interpretation-by-distillation formulation that is defined relative to other ML models. We evaluate our proposed framework on the MNIST, Fashion-MNIST and Stanford40 datasets.
  arXiv  Detail & Related papers  (2020-12-05T17:42:50Z)
• Dif-MAML: Decentralized Multi-Agent Meta-Learning [54.39661018886268]
  We propose a cooperative multi-agent meta-learning algorithm, referred to as MAML or Dif-MAML. We show that the proposed strategy allows a collection of agents to attain agreement at a linear rate and to converge to a stationary point of the aggregate MAML. Simulation results illustrate the theoretical findings and the superior performance relative to the traditional non-cooperative setting.
  arXiv  Detail & Related papers  (2020-10-06T16:51:09Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)
• Explainability in Deep Reinforcement Learning [68.8204255655161]
  We review recent works in the direction to attain Explainable Reinforcement Learning (XRL) In critical situations where it is essential to justify and explain the agent's behaviour, better explainability and interpretability of RL models could help gain scientific insight on the inner workings of what is still considered a black box.
  arXiv  Detail & Related papers  (2020-08-15T10:11:42Z)
• The Minimum Description Length Principle for Pattern Mining: A Survey [0.07614628596146598]
  This describes the Minimum Description Length principle applied to pattern mining. After giving an outline of relevant concepts from information theory and coding, we review MDL-based methods for mining various types of data and patterns.
  arXiv  Detail & Related papers  (2020-07-28T06:24:39Z)
• A Chain Graph Interpretation of Real-World Neural Networks [58.78692706974121]
  We propose an alternative interpretation that identifies NNs as chain graphs (CGs) and feed-forward as an approximate inference procedure. The CG interpretation specifies the nature of each NN component within the rich theoretical framework of probabilistic graphical models. We demonstrate with concrete examples that the CG interpretation can provide novel theoretical support and insights for various NN techniques.
  arXiv  Detail & Related papers  (2020-06-30T14:46:08Z)
• On the Information Plane of Autoencoders [0.0]
  Information Plane (IP) was proposed to analyze hidden layers in deep learning networks. In this work, we derive a theoretical convergence for the IP of autoencoders. Our theoretical IP for autoencoders could be used as a benchmark to validate new methods to estimate MI in neural networks.
  arXiv  Detail & Related papers  (2020-05-15T21:05:49Z)
• Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G Networks [84.2155885234293]
  We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC. To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
  arXiv  Detail & Related papers  (2020-02-22T14:38:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.