Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development

• URL: http://arxiv.org/abs/2606.07207v1
• Date: Fri, 05 Jun 2026 12:19:00 GMT
• Title: Entropy as a Structural Prior: How a Log-Barrier on DiT Belief Space Drives Musical Diversity and Development
• Abstract summary: Confidence-based loss weighting is usually avoided in generative models because it accelerates errors when the model is confidently wrong.<n>We introduce the Eisbach log-barrier, a parameter-free weight derived from the entropy of the DiT output's spatial energy distribution.<n>It unexpectedly yields stronger thematic development, clearer acoustic differentiation, and higher textural diversity than unweighted training.
• Score: 0.13750624267664155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Confidence-based loss weighting is usually avoided in generative models because it accelerates errors when the model is confidently wrong, but this intuition breaks down in supervised diffusion training. We introduce the Eisbach log-barrier, a parameter-free weight derived from the entropy of the DiT output's spatial energy distribution: high entropy damps the gradient, while low entropy preserves it. Applied to LoRA fine-tuning of Stable Audio 3 Medium on MusicCaps, it unexpectedly yields stronger thematic development, clearer acoustic differentiation, and higher textural diversity than unweighted training, the opposite of mode collapse. This works because in supervised diffusion the gradient direction is locked to ground truth, so confidence only scales the step size, and because temporal entropy downweights flat samples while preserving high-contrast ones. The result is an online, self-referential data curriculum that emerges purely from the forward pass, with analyzed noise-level dynamics and testable predictions.

Related papers

• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics [92.39053980710702]
  Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion.<n>This approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate.<n>We propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:40Z)
• Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansion [1.7770888723114693]
  We show that fine-tuning LLMs to near-zero training loss enhances open-ended generation quality and mitigates repetition in greedy decoding.<n>We introduce Late-Stage LoRA, a targeted fine-tuning strategy that updates only the final 5 layers, yielding robust generation with minimal parameter updates.
  arXiv  Detail & Related papers  (2026-05-21T14:52:48Z)
• Provably Learning Diffusion Models under the Manifold Hypothesis: Collapse and Refine [60.669081685261965]
  Diffusion models generate high-dimensional data with remarkable quality.<n>How their training efficiently learns the score function remains theoretically unexplained.<n>We instantiate this principle as Score-induced Latent Diffusion (SiLD)
  arXiv  Detail & Related papers  (2026-05-16T16:51:10Z)
• Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks [74.46751089984072]
  gradient descent (SGD) noise on descent in deep linear networks is poorly understood.<n>We investigate the dynamics of SGD during training of DLNs in the saddle-to-saddle regime.<n>Our results establish that SGD noise encodes information about the progression of feature learning but does not fundamentally alter the saddle-to-saddle dynamics.
  arXiv  Detail & Related papers  (2026-04-07T18:43:08Z)
• EDIS: Diagnosing LLM Reasoning via Entropy Dynamics [3.858418431840288]
  We show that the emphtemporal evolution of confidence during generation carries richer information than aggregate statistics alone.<n>We introduce the Entropy Dynamics Instability Score (textbfEDIS), a trajectory-level metric quantifying instability in entropy evolution.
  arXiv  Detail & Related papers  (2026-02-01T15:43:50Z)
• Entropy Production in Machine Learning Under Fokker-Planck Probability Flow [0.0]
  We propose an entropy-based retraining framework grounded in non-equilibrium cost dynamics.<n>We show that entropy-triggered retraining achieves predictive performance comparable to high-frequency retraining.
  arXiv  Detail & Related papers  (2026-01-02T04:01:57Z)
• Entropy-Reservoir Bregman Projection: An Information-Geometric Unification of Model Collapse [3.533187668612022]
  We present EntropyReser Bregman Projection- ERBP, an information-geometric framework that unifies these phenomena.<n>Our theory yields a necessary condition for collapse, (ii) a sufficient condition that guarantees a non-language entropy floor, and (iii) closed-form rates that depend on sample size.
  arXiv  Detail & Related papers  (2025-12-16T19:50:03Z)
• Diffusion Models are Molecular Dynamics Simulators [0.0]
  We prove that a denoising diffusion sampler equipped with a sequential bias across the batch dimension is exactly an Euler-Maruyama integrator for overdamped Langevin dynamics.<n>Each reverse denoising step, with its associated spring stiffness, can be interpreted as one step of a differential equation with an effective time step set jointly by the noise schedule and that stiffness.<n>The learned score then plays the role of the gradient of a learned energy, yielding a precise correspondence between diffusion sampling and Langevin time evolution.
  arXiv  Detail & Related papers  (2025-11-21T19:48:32Z)
• Dynamical Diffusion: Learning Temporal Dynamics with Diffusion Models [71.63194926457119]
  We introduce Dynamical Diffusion (DyDiff), a theoretically sound framework that incorporates temporally aware forward and reverse processes.<n>Experiments across scientifictemporal forecasting, video prediction, and time series forecasting demonstrate that Dynamical Diffusion consistently improves performance in temporal predictive tasks.
  arXiv  Detail & Related papers  (2025-03-02T16:10:32Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.