A Geometrically-Grounded Drive for MDL-Based Optimization in Deep Learning

• URL: http://arxiv.org/abs/2603.12304v1
• Date: Thu, 12 Mar 2026 08:31:00 GMT
• Title: A Geometrically-Grounded Drive for MDL-Based Optimization in Deep Learning
• Abstract summary: This paper introduces a novel optimization framework that fundamentally integrates the Minimum Description Length (MDL) principle into the training dynamics of deep neural networks.<n>We provide a practical, computationally efficient algorithm with $O(N log N)$ per-iteration complexity (Theoremrefthm:complexity), alongside guarantees for numerical stability (Theoremrefthm:stability) and exponential convergence under convexity assumptions (Theoremrefthm:convergence_rate)<n>This work provides a principled path toward more autonomous, generalizable, and interpretable AI
• Score: 3.2452107817263003
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: This paper introduces a novel optimization framework that fundamentally integrates the Minimum Description Length (MDL) principle into the training dynamics of deep neural networks. Moving beyond its conventional role as a model selection criterion, we reformulate MDL as an active, adaptive driving force within the optimization process itself. The core of our method is a geometrically-grounded cognitive manifold whose evolution is governed by a \textit{coupled Ricci flow}, enriched with a novel \textit{MDL Drive} term derived from first principles. This drive, modulated by the task-loss gradient, creates a seamless harmony between data fidelity and model simplification, actively compressing the internal representation during training. We establish a comprehensive theoretical foundation, proving key properties including the monotonic decrease of description length (Theorem~\ref{thm:convergence}), a finite number of topological phase transitions via a geometric surgery protocol (Theorems~\ref{thm:surgery}, \ref{thm:ultimate_fate}), and the emergence of universal critical behavior (Theorem~\ref{thm:universality}). Furthermore, we provide a practical, computationally efficient algorithm with $O(N \log N)$ per-iteration complexity (Theorem~\ref{thm:complexity}), alongside guarantees for numerical stability (Theorem~\ref{thm:stability}) and exponential convergence under convexity assumptions (Theorem~\ref{thm:convergence_rate}). Empirical validation on synthetic regression and classification tasks confirms the theoretical predictions, demonstrating the algorithm's efficacy in achieving robust generalization and autonomous model simplification. This work provides a principled path toward more autonomous, generalizable, and interpretable AI systems by unifying geometric deep learning with information-theoretic principles.

Related papers

• INSPIRE: An Internalize-Then-Improve Approach for Example-Driven Mathematical Reasoning [61.60016491769943]
  In human mathematics education, example-based reasoning such as constructing counterexamples to test theorem boundaries reflects deep conceptual understanding.<n>We propose INSPIRE, an Internalize-Then-Improve approach combining Reference-Guided Student Internalization (RGSI) with a stage-wise preference training strategy.<n>Experiments across multiple model scales and families demonstrate consistent improvements, even surpassing larger open-source models.
  arXiv  Detail & Related papers  (2026-08-27T03:24:03Z)
• Anisotropic Modality Align [91.23979617826926]
  Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data.<n>Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data.<n>The core obstacle lies in the persistent Modality Gap in the shared space.
  arXiv  Detail & Related papers  (2026-05-08T14:53:24Z)
• Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective [8.18887614959892]
  A growing body of recent work demonstrates the successful application of ZO methods to fine-tuning Large Language Models.<n>We derive the one-step learning dynamics of ZO SGD, where the empirical Neural Tangent Kernel (eNTK) emerges as the key term governing the learning behavior.<n>We believe that this kernel-based framework offers a novel perspective for understanding ZO methods within the context of learning dynamics.
  arXiv  Detail & Related papers  (2026-05-05T05:28:29Z)
• Why Self-Rewarding Works: Theoretical Guarantees for Iterative Alignment of Language Models [50.248686344277246]
  Self-Rewarding Language Models (SRLMs) achieve notable success in iteratively improving alignment without external feedback.<n>This paper provides the first rigorous theoretical guarantees for SRLMs.
  arXiv  Detail & Related papers  (2026-01-30T03:45:43Z)
• Deep Unfolding: Recent Developments, Theory, and Design Guidelines [99.63555420898554]
  This article provides a tutorial-style overview of deep unfolding, a framework that transforms optimization algorithms into structured, trainable ML architectures.<n>We review the foundations of optimization for inference and for learning, introduce four representative design paradigms for deep unfolding, and discuss the distinctive training schemes that arise from their iterative nature.
  arXiv  Detail & Related papers  (2025-12-03T13:16:35Z)
• Learning Geometry: A Framework for Building Adaptive Manifold Models through Metric Optimization [8.201374511929538]
  This paper proposes a novel paradigm for machine learning that moves beyond traditional parameter optimization.<n>We optimize the metric tensor field on a manifold with a predefined topology, thereby dynamically shaping the geometric structure of the model space.<n>This work lays a solid foundation for constructing fully dynamic "meta-learners" capable of autonomously evolving their geometry and topology.
  arXiv  Detail & Related papers  (2025-10-30T01:53:32Z)
• Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers [12.400454043294296]
  This paper introduces the theoretical notion of universalityally optimal description length objectives.<n>We show that such objectives can be tractable and differentiable by constructing and analyzing a variational objective.<n>More broadly, by providing a theoretical framework for identifying description length objectives with strong algorithmic guarantees, we outline a potential path towards training neural networks that achieve greater compression and generalization.
  arXiv  Detail & Related papers  (2025-09-26T15:02:24Z)
• Geometric Meta-Learning via Coupled Ricci Flow: Unifying Knowledge Representation and Quantum Entanglement [7.410691988131121]
  This paper establishes a unified framework integrating geometric flows with deep learning through three fundamental innovations.<n>First, we propose a thermodynamically coupled Ricci flow that dynamically adapts parameter space geometry to loss landscape topology.<n>Second, we derive explicit phase transition thresholds and critical learning rates through curvature blowup analysis.<n>Third, we establish an AdS/CFT-type holographic duality (Theoremrefthm:ads) between neural networks and conformal field theories.
  arXiv  Detail & Related papers  (2025-03-25T17:32:31Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• PAC-Chernoff Bounds: Understanding Generalization in the Interpolation Regime [6.645111950779666]
  This paper introduces a distribution-dependent PAC-Chernoff bound that exhibits perfect tightness for interpolators.<n>We present a unified theoretical framework revealing why certain interpolators show an exceptional generalization, while others falter.
  arXiv  Detail & Related papers  (2023-06-19T14:07:10Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Structured Optimal Variational Inference for Dynamic Latent Space Models [16.531262817315696]
  We consider a latent space model for dynamic networks, where our objective is to estimate the pairwise inner products plus the intercept of the latent positions. To balance posterior inference and computational scalability, we consider a structured mean-field variational inference framework.
  arXiv  Detail & Related papers  (2022-09-29T22:10:42Z)
• The Dynamics of Riemannian Robbins-Monro Algorithms [101.29301565229265]
  We propose a family of Riemannian algorithms generalizing and extending the seminal approximation framework of Robbins and Monro. Compared to their Euclidean counterparts, Riemannian algorithms are much less understood due to lack of a global linear structure on the manifold. We provide a general template of almost sure convergence results that mirrors and extends the existing theory for Euclidean Robbins-Monro schemes.
  arXiv  Detail & Related papers  (2022-06-14T12:30:11Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• Stochastic Flows and Geometric Optimization on the Orthogonal Group [52.50121190744979]
  We present a new class of geometrically-driven optimization algorithms on the orthogonal group $O(d)$. We show that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, flows and metric learning.
  arXiv  Detail & Related papers  (2020-03-30T15:37:50Z)
• Formal Synthesis of Lyapunov Neural Networks [61.79595926825511]
  We propose an automatic and formally sound method for synthesising Lyapunov functions. We employ a counterexample-guided approach where a numerical learner and a symbolic verifier interact to construct provably correct Lyapunov neural networks. Our method synthesises Lyapunov functions faster and over wider spatial domains than the alternatives, yet providing stronger or equal guarantees.
  arXiv  Detail & Related papers  (2020-03-19T17:21:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.