From Objectives to What Models Learn: A Landau Theory of Invariant Learning

• URL: http://arxiv.org/abs/2608.09396v1
• Date: Mon, 10 Aug 2026 10:22:10 GMT
• Title: From Objectives to What Models Learn: A Landau Theory of Invariant Learning
• Abstract summary: Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque.<n>We address this objective-behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives.<n>In one- and two-hidden-layer ReLU networks, the same signatures remain predictive of qualitative regularization-path behavior despite depth-dependent shifts in scale.
• Score: 13.862849637472076
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Invariant learning seeks representations that remain predictive across environments, yet the behavior of its objectives along the regularization path is often opaque. We address this objective-behavior gap by viewing representation learning as multimode magnetization and deriving, from concrete invariant-learning objectives, a Landau-type effective free energy whose low-order coefficients form objective signatures and induce distinct regularization phenotypes. Effective quadratic corrections move the phase boundary and enable finite-strength mode elimination; quartic corrections regulate post-onset amplitude and typically leave residual loading at finite strength; higher-order structure governs non-monotone tails, instability, and collapse at large regularization. In a canonical bilinear model, the theory yields closed-form phase boundaries and steady-state loadings, as well as distinct critical strengths for shortcut and stable modes that define a selective-retention window. Controlled experiments confirm the predicted phase boundaries, loadings, and regularization phenotypes. In one- and two-hidden-layer ReLU networks, the same signatures remain predictive of qualitative regularization-path behavior despite depth-dependent shifts in scale. A matrix extension generalizes the framework to coupled collective modes and yields a spectral phase-boundary criterion. Together, the framework turns low-order objective signatures into predictions of regularization phenotypes and, ultimately, of what models learn as regularization varies.

Related papers

• Finite-Sample Metric Non-Collapse for Geometrically Supervised Latent World Models in Control [0.0]
  We establish a finite-sample learning-to-control theory for geometrically supervised latent models of nonlinear deterministic systems.<n>We introduce an encoder-only local--global metric hinge that enforces directional resolution and separated-state discrimination.<n>A modular deterministic transfers the learned certificates to trajectory, finite-horizon cost, learned-cost-head, and corollary guarantees, while a validated finite-net result enables sharper model-specific certification.
  arXiv  Detail & Related papers  (2026-08-07T14:25:33Z)
• GatedLinear: Adaptive Routing of Complementary Linear Bases for Time Series Forecasting [11.155508081323694]
  Time series forecasting requires models to capture diverse, often mutually exclusive, temporal dynamics.<n>Recent deep learning models have improved accuracy, but they typically force these diverse patterns through a single computational backbone.<n>We propose GatedLinear: a lightweight framework that frames forecasting as the adaptive routing of complementary linear bases.
  arXiv  Detail & Related papers  (2026-07-10T15:42:01Z)
• Predictable GRPO: A Closed-Form Model of Training Dynamics [1.0537323885285492]
  We develop a first-principles reduced-order model of these dynamics.<n>Group size enters, to leading order, as a noise temperature.<n>Across three models and two group sizes, the closed-form trajectory fits training reward to $R2 geq 0.91$.
  arXiv  Detail & Related papers  (2026-06-29T18:19:09Z)
• Preventing the Breakdown of Tight-Binding Waveguide Optics by Löwdin Orthogonalization [0.0]
  We study the dynamics of the total light field by examining the coupling between the guided modes of individual single-mode substructures.<n>We introduce a modified TB framework based on the Lwdin orthogonalization, which constructs an orthonormal basis from the non-orthogonal guided modes.<n>The resulting Lwdin-TB method restores the standard eigenvalue problem and yields excellent agreement with exact beam propagation simulations.
  arXiv  Detail & Related papers  (2026-05-29T09:43:07Z)
• Probability-Conserving Flow Guidance [49.03107678763765]
  Adaptive Manifold Guidance (AdaMaG) improves realism, reduces hallucinations, and controlled desaturation in high-guidance.<n>Across image generation benchmarks, AdaMaG improves realism, reduces hallucinations, and controlled desaturation in high-guidance.
  arXiv  Detail & Related papers  (2026-05-19T16:34:01Z)
• Explicit Dropout: Deterministic Regularization for Transformer Architectures [55.09895958546215]
  Dropout is a widely used regularization technique in deep learning, but its effects are typically realized through masking.<n>We propose a deterministic formulation that expresses dropout as an additive regularizer directly incorporated into the training loss.
  arXiv  Detail & Related papers  (2026-04-22T12:45:51Z)
• Deep Sequence Modeling with Quantum Dynamics: Language as a Wave Function [0.0]
  We introduce a sequence modeling framework in which the latent state is a complex-valued wave function evolving on a finite-dimensional Hilbert space under a learned, time-dependent Hamiltonian.<n> Token probabilities are extracted using the Born rule, a quadratic measurement operator that couples magnitudes and relative phases.<n>We derive a continuity equation for the latent probability mass, yielding conserved pairwise currents that serve as a built-in diagnostic.
  arXiv  Detail & Related papers  (2026-02-24T23:42:18Z)
• Gradients Must Earn Their Influence: Unifying SFT with Generalized Entropic Objectives [22.29000001610794]
  Standard negative log-likelihood for Supervised Fine-Tuning (SFT) applies uniform token-level weighting.<n>This rigidity creates a two-fold failure mode: (i) overemphasizing low-probability targets can amplify gradients on noisy supervision and disrupt robust priors, and (ii) uniform weighting provides weak sharpening when the model is already confident.<n>Existing methods fail to resolve the resulting plasticity--stability dilemma, often suppressing necessary learning signals alongside harmful ones.<n>We introduce Dynamic Entropy Fine-Tuning (DEFT), a parameter-free objective that modulates the
  arXiv  Detail & Related papers  (2026-02-11T22:56:43Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• The Procrustean Bed of Time Series: The Optimization Bias of Point-wise Loss [53.542743390809356]
  This paper aims to provide a first-principles analysis of the Expectation of Optimization Bias (EOB)<n>Our analysis reveals a fundamental paradigm paradox: the more deterministic and structured the time series, the more severe the bias by point-wise loss function.<n>We present a concrete solution that simultaneously achieves both principles via DFT or DWT.
  arXiv  Detail & Related papers  (2025-12-21T06:08:22Z)
• Solving Inverse Problems with FLAIR [68.87167940623318]
  We present FLAIR, a training-free variational framework that leverages flow-based generative models as prior for inverse problems.<n>Results on standard imaging benchmarks demonstrate that FLAIR consistently outperforms existing diffusion- and flow-based methods in terms of reconstruction quality and sample diversity.
  arXiv  Detail & Related papers  (2025-06-03T09:29:47Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.