Related papers: Latent States in Neural Networks: Recovering the Temporal Structure of Drifting Data from Model Weights

Latent States in Neural Networks: Recovering the Temporal Structure of Drifting Data from Model Weights

URL: http://arxiv.org/abs/2607.27482v1
Date: Wed, 29 Jul 2026 21:51:29 GMT
Title: Latent States in Neural Networks: Recovering the Temporal Structure of Drifting Data from Model Weights
Abstract summary: A temporally drifting data stream may pass through discrete regimes rather than changing continuously.<n>We ask whether such regimes are recoverable from the weights of models trained on the stream using a hidden Markov model (HMM)<n>We study this question in two domains known to drift over time: multimodal misinformation detection, using the Fakeddit dataset; and sentiment analysis, using the Yelp dataset.
Score: 1.7767466724342065
License: http://creativecommons.org/licenses/by/4.0/
Abstract: A temporally drifting data stream may pass through discrete regimes rather than changing continuously. We ask whether such regimes are recoverable from the weights of models trained on the stream, using a hidden Markov model (HMM) fit to the chronologically ordered trajectory of those weights. We study this question in two domains known to drift over time: multimodal misinformation detection, using the Fakeddit dataset; and sentiment analysis, using the Yelp dataset. We train classifiers on consecutive temporal windows and fit an HMM to the trajectory of their aligned weights, recovering latent states that partition each timeline into coherent phases. On both datasets, classifiers generalize better to data from windows sharing the state of their training window than to windows across state boundaries. This within-state transfer advantage survives a control for temporal proximity and modestly exceeds the advantage recovered by a naive partition into contiguous states of equal size. Although the states are estimated solely from model weights, they correlate more strongly with shifts in the data's class distribution than with the weight-space geometry used to estimate them. After class divergence and lag are residualized out, the within-state advantage exceeds its permutation null on both tasks, indicating that the states recover structure relevant to transfer beyond the data distribution. Every effect replicates on both tasks but is attenuated on Yelp, whose label distribution is more temporally stable.

Related papers

QuantFlow: A Federated Mamba-Based Post-Transformer Foundation Model for Time-Series Forecasting [0.0]
Time-series forecasting supports decisions in finance, en-ergy, transportation, public health, and industrial monitoring.<n>Recent foundation models improve transfer across forecast-ing tasks, but many depend on centralized data and Trans-former attention.<n>This paper presents QuantFlow, a probabilistic forecasting framework that com-bines inverted sequence embedding, bidirectional Mamba state-space decoders, quantile regression, and federated learning.
arXiv Detail & Related papers (2026-07-02T14:16:48Z)
Stochastic Lifting for Generating Trajectories of Stochastic Physical Systems [12.19832939917878]
transition from current state to next state can often be modeled as the combination of a smooth map and an explicit source of randomness.<n>This paper exploits this structure by attaching an independent, high-dimensional random label to each state transition in the training data and fitting a transition map from the current state and label to the next state using a standard regression loss.<n>Fresh labels are sampled at each time step and the learned map is rolled forward autoregressively, generating diverse trajectories with a single network evaluation per time step.
arXiv Detail & Related papers (2026-05-28T00:10:45Z)
ForcingDAS: Unified and Robust Data Assimilation via Diffusion Forcing [24.41363445015799]
We introduce ForcingDAS, a unified and robust data assimilation framework.<n>ForcingDAS learns a joint-trajectory prior instead of frame-to-frame transitions.<n>We evaluate ForcingDAS on 2D Navier-Stokes vorticity, precipitation nowcasting, and global atmospheric state estimation.
arXiv Detail & Related papers (2026-05-14T02:34:33Z)
Posterior Augmented Flow Matching [64.1559809786948]
Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
arXiv Detail & Related papers (2026-05-01T17:59:59Z)
STDiff: A State Transition Diffusion Framework for Time Series Imputation in Industrial Systems [2.3895981099137535]
We propose STDiff, which reframes imputation as learning how the system evolves from one state to the next.<n> STDiff consistently achieves the lowest errors, with its advantage increasing for longer gaps.<n>These results support dynamics-aware, explicitly conditioned imputation as a robust approach for industrial time series.
arXiv Detail & Related papers (2025-08-26T13:14:53Z)
OmniTraj: Pre-Training on Heterogeneous Data for Adaptive and Zero-Shot Human Trajectory Prediction [62.385417528148224]
We present OmniTraj, a Transformer-based model pre-trained on a large-scale, heterogeneous dataset.<n>Experiments show that explicitly conditioning on the frame rate enables OmniTraj to achieve state-of-the-art zero-shot transfer performance.
arXiv Detail & Related papers (2025-07-31T15:37:09Z)
LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised Time Series Anomaly Detection [49.52429991848581]
We propose a Light and Anti-overfitting Retraining Approach (LARA) for deep variational auto-encoder based time series anomaly detection methods (VAEs) This work aims to make three novel contributions: 1) the retraining process is formulated as a convex problem and can converge at a fast rate as well as prevent overfitting; 2) designing a ruminate block, which leverages the historical data without the need to store them; and 3) mathematically proving that when fine-tuning the latent vector and reconstructed data, the linear formations can achieve the least adjusting errors between the ground truths and the fine-tuned ones.
arXiv Detail & Related papers (2023-10-09T12:36:16Z)
Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
arXiv Detail & Related papers (2023-02-17T18:45:04Z)
Averaging Spatio-temporal Signals using Optimal Transport and Soft Alignments [110.79706180350507]
We show that our proposed loss can be used to define temporal-temporal baryechecenters as Fr'teche means duality. Experiments on handwritten letters and brain imaging data confirm our theoretical findings.
arXiv Detail & Related papers (2022-03-11T09:46:22Z)
How Well Do Sparse Imagenet Models Transfer? [75.98123173154605]
Transfer learning is a classic paradigm by which models pretrained on large "upstream" datasets are adapted to yield good results on "downstream" datasets. In this work, we perform an in-depth investigation of this phenomenon in the context of convolutional neural networks (CNNs) trained on the ImageNet dataset. We show that sparse models can match or even outperform the transfer performance of dense models, even at high sparsities.
arXiv Detail & Related papers (2021-11-26T11:58:51Z)
Dynamical Wasserstein Barycenters for Time-series Modeling [16.212262513825717]
Most prior work assumes instantaneous transitions between pure discrete states. We propose a Wasserstein barycentric (DWB) model that estimates the system state over time. Experiments on several human activity datasets show that our proposed DWB model accurately learns the generating distribution of pure states.
arXiv Detail & Related papers (2021-10-13T14:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.