Interpreting learning dynamics of autoencoders: Transient scaling and emerging concepts of the Ising model

• URL: http://arxiv.org/abs/2607.10285v1
• Date: Sat, 11 Jul 2026 12:31:34 GMT
• Title: Interpreting learning dynamics of autoencoders: Transient scaling and emerging concepts of the Ising model
• Abstract summary: unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables.<n>We quantify learning across multiple spatial scales and reveal two distinct dynamical regimes.<n>A dynamical viewpoint of learning is established in which intrinsic properties expose the effects of forced changes in representation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study how unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables underlying the data-generating process. Without embedding domain knowledge, we mimic a typical discovery setting: We quantify learning across multiple spatial (coarse-graining) scales and reveal two distinct dynamical regimes controlled by main hyperparameters (model depth, width, and learning rate) -- a magnetization-dominated regime and an energy-dominated regime characterized by trade-offs in their representation quality. The first regime is a transitory state exhibiting dynamical scaling and fluctuations that follow an ordering-to-scale; the second gradually shifts resolution towards smaller scales relevant for the energy representation. Deep models trained at moderate and fast rates become arrested before reaching these regimes. With a novel analysis of recursive-dynamic trajectories, we demonstrate that prediction errors induce flow fields that produce a common trajectory topology across all representation spaces. A dynamical viewpoint of learning is established in which intrinsic properties expose the effects of forced changes in representation during training. We utilize the intuition that learning operates as a process driven far from equilibrium by fluctuations from the training data and optimizer to provide an interpretive basis grounded in both the physical world and the machine models that represent it.

Related papers

• Generalizing Dynamics Modeling More Easily from Representation Perspective [17.958624542632947]
  We conduct a generalized Pre-trained Dynamics EncoDER (PDEDER)<n>PDEDER embeds the original state observations into a latent space where the dynamics can be captured more easily.<n>We evaluate PDEDER on 12 dynamic systems by short/long-term forecasting under both in-domain and cross-domain settings.
  arXiv  Detail & Related papers  (2026-03-24T00:15:47Z)
• Transformers for dynamical systems learn transfer operators in-context [4.211463778317431]
  In-context learning challenges conventional understanding of learning and adaptation in physical systems.<n>We train a small two-layer, single-head transformer to forecast one dynamical system, and then evaluate its ability to forecast a different dynamical system without retraining.<n>We discover that attention-based models apply a transfer-operator forecasting strategy in-context.
  arXiv  Detail & Related papers  (2026-02-21T01:03:15Z)
• Towards Physically Interpretable World Models: Meaningful Weakly Supervised Representations for Visual Trajectory Prediction [0.1534667887016089]
  We propose a novel architecture that aligns learned latent representations with real-world physical quantities.<n>Three case studies demonstrate that our approach achieves physical interpretability and accurate state predictions.
  arXiv  Detail & Related papers  (2024-12-17T12:51:24Z)
• Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis [97.54180451650122]
  We study the dynamics of training a shallow transformer on a task of recognizing co-occurrence of two designated words. We analyze the gradient flow dynamics of simultaneously training three attention matrices and a linear layer. We prove a novel property of the gradient flow, termed textitautomatic balancing of gradients, which enables the loss values of different samples to decrease almost at the same rate and further facilitates the proof of near minimum training loss.
  arXiv  Detail & Related papers  (2024-10-12T17:50:58Z)
• Dynamical Mean-Field Theory of Self-Attention Neural Networks [0.0]
  Transformer-based models have demonstrated exceptional performance across diverse domains. Little is known about how they operate or what are their expected dynamics. We use methods for the study of asymmetric Hopfield networks in nonequilibrium regimes.
  arXiv  Detail & Related papers  (2024-06-11T13:29:34Z)
• In-Context Convergence of Transformers [63.04956160537308]
  We study the learning dynamics of a one-layer transformer with softmax attention trained via gradient descent. For data with imbalanced features, we show that the learning dynamics take a stage-wise convergence process.
  arXiv  Detail & Related papers  (2023-10-08T17:55:33Z)
• Latent State Models of Training Dynamics [51.88132043461152]
  We train models with different random seeds and compute a variety of metrics throughout training. We then fit a hidden Markov model (HMM) over the resulting sequences of metrics. We use the HMM representation to study phase transitions and identify latent "detour" states that slow down convergence.
  arXiv  Detail & Related papers  (2023-08-18T13:20:08Z)
• Understanding Self-Predictive Learning for Reinforcement Learning [61.62067048348786]
  We study the learning dynamics of self-predictive learning for reinforcement learning. We propose a novel self-predictive algorithm that learns two representations simultaneously.
  arXiv  Detail & Related papers  (2022-12-06T20:43:37Z)
• Physics-Inspired Temporal Learning of Quadrotor Dynamics for Accurate Model Predictive Trajectory Tracking [76.27433308688592]
  Accurately modeling quadrotor's system dynamics is critical for guaranteeing agile, safe, and stable navigation. We present a novel Physics-Inspired Temporal Convolutional Network (PI-TCN) approach to learning quadrotor's system dynamics purely from robot experience. Our approach combines the expressive power of sparse temporal convolutions and dense feed-forward connections to make accurate system predictions.
  arXiv  Detail & Related papers  (2022-06-07T13:51:35Z)
• Decomposed Linear Dynamical Systems (dLDS) for learning the latent components of neural dynamics [6.829711787905569]
  We propose a new decomposed dynamical system model that represents complex non-stationary and nonlinear dynamics of time series data. Our model is trained through a dictionary learning procedure, where we leverage recent results in tracking sparse vectors over time. In both continuous-time and discrete-time instructional examples we demonstrate that our model can well approximate the original system.
  arXiv  Detail & Related papers  (2022-06-07T02:25:38Z)
• Bridging Physics-based and Data-driven modeling for Learning Dynamical Systems [36.04381259949935]
  We formalize the learning of physics-based models as AutoODE, which leverages automatic differentiation to estimate the model parameters. Our method obtains a 57.4% reduction in mean absolute errors for 7-day ahead COVID-19 forecasting compared with the best deep learning competitor.
  arXiv  Detail & Related papers  (2020-11-20T20:16:10Z)
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning [124.9856253431878]
  We decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.
  arXiv  Detail & Related papers  (2020-05-14T08:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.