Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift

• URL: http://arxiv.org/abs/2607.05908v1
• Date: Tue, 07 Jul 2026 07:03:03 GMT
• Title: Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift
• Abstract summary: Real-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems.<n>We present a systematic empirical comparison of temporal robustness across three heterogeneous, time-indexed domains.<n>Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer-based encoders.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems. However, the extent to which architectural choices and their associated inductive biases affect temporal robustness remains insufficiently understood. We present a systematic empirical comparison of temporal robustness across three heterogeneous, time-indexed domains encompassing image classification, multi-label text classification, and text regression tasks. Using a unified evaluation framework based on temporal drift matrices, we train models on cumulative historical data and evaluate their performance on both earlier and later time periods, thereby quantifying cross-temporal generalization. Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer-based encoders. Collectively, the results show that architectural inductive biases systematically shape temporal robustness: models whose inductive biases lead them to exploit localized, highly discriminative features attain the highest in-distribution accuracy, yet those features are often the ones that change most over time, so these models degrade fastest, while pretrained encoders that draw on coarser, more stable representations drift more gradually. These observations offer practical guidance for selecting architectures for real-world systems subject to temporal drift.

Related papers

• Symbolic Neural ODEs: Learning interpretable models from time-series data [0.0]
  We present a machine learning framework for identifying sparse, interpretable models of dynamical systems from time-series data.<n>Our approach parameterizes the underlying vector field using a neural architecture and trains it by minimizing a multi-step prediction loss over a finite horizon.<n>We show that multi-step training yields models with accurate short-term dynamics and strong agreement in long-time statistical properties.
  arXiv  Detail & Related papers  (2026-08-22T21:31:07Z)
• SPaRSe-TIME: Saliency-Projected Low-Rank Temporal Modeling for Efficient and Interpretable Time Series Prediction [0.0]
  SPaRSe-TIME is a framework that models time series through a decomposition into three complementary components: saliency, memory, and trend.<n>Experiments on diverse real-world datasets demonstrate that SPaRSe-TIME achieves competitive performance.<n>Overall, SPaRSe-TIME offers a principled alternative to monolithic sequence models, bridging efficiency, interpretability, and performance, and providing a scalable framework for time series learning.
  arXiv  Detail & Related papers  (2026-04-19T09:55:28Z)
• Seeking SOTA: Time-Series Forecasting Must Adopt Taxonomy-Specific Evaluation to Dispel Illusory Gains [17.650604674194984]
  We argue that the current practice of evaluating AI/ML time-series forecasting models obscures real progress by overlooking the performance of efficient classical methods.<n>We demonstrate that these "standard" datasets often exhibit dominant autocorrelation patterns and seasonal cycles that can be effectively captured by simpler linear or statistical models.
  arXiv  Detail & Related papers  (2026-03-16T16:34:26Z)
• MEMTS: Internalizing Domain Knowledge via Parameterized Memory for Retrieval-Free Domain Adaptation of Time Series Foundation Models [51.506429027626005]
  Memory for Time Series (MEMTS) is a lightweight and plug-and-play method for retrieval-free domain adaptation in time series forecasting.<n>Key component of MEMTS is a Knowledge Persistence Module (KPM), which internalizes domain-specific temporal dynamics.<n>This paradigm shift enables MEMTS to achieve accurate domain adaptation with constant-time inference and near-zero latency.
  arXiv  Detail & Related papers  (2026-02-14T14:00:06Z)
• Learning Time in Static Classifiers [44.358377952850994]
  We propose a simple yet effective framework that equips standard feedforward classifiers with temporal reasoning.<n>We use a novel Support-Exemplar-Query (SEQ) learning paradigm, which structures training data into temporally coherent trajectories.<n>Our approach bridges static and temporal learning in a modular and data-efficient manner, requiring only a simple on top of pre-extracted features.
  arXiv  Detail & Related papers  (2025-11-15T18:42:51Z)
• Multivariate Long-term Time Series Forecasting with Fourier Neural Filter [42.60778405812048]
  We introduce FNF as the backbone and DBD as architecture to provide excellent learning capabilities and optimal learning pathways for spatial-temporal modeling.<n>We show that FNF unifies local time-domain and global frequency-domain information processing within a single backbone that extends naturally to spatial modeling.
  arXiv  Detail & Related papers  (2025-06-10T18:40:20Z)
• In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention [52.159541540613915]
  We study how multi-head softmax attention models are trained to perform in-context learning on linear data.<n>Our results reveal that in-context learning ability emerges from the trained transformer as an aggregated effect of its architecture and the underlying data distribution.
  arXiv  Detail & Related papers  (2025-03-17T02:00:49Z)
• Powerformer: A Transformer with Weighted Causal Attention for Time-series Forecasting [50.298817606660826]
  We introduce Powerformer, a novel Transformer variant that replaces noncausal attention weights with causal weights that are reweighted according to a smooth heavy-tailed decay.<n>Our empirical results demonstrate that Powerformer achieves state-of-the-art accuracy on public time-series benchmarks.<n>Our analyses show that the model's locality bias is amplified during training, demonstrating an interplay between time-series data and power-law-based attention.
  arXiv  Detail & Related papers  (2025-02-10T04:42:11Z)
• Detecting Anomalies in Dynamic Graphs via Memory enhanced Normality [39.476378833827184]
  Anomaly detection in dynamic graphs presents a significant challenge due to the temporal evolution of graph structures and attributes. We introduce a novel spatial- temporal memories-enhanced graph autoencoder (STRIPE) STRIPE significantly outperforms existing methods with 5.8% improvement in AUC scores and 4.62X faster in training time.
  arXiv  Detail & Related papers  (2024-03-14T02:26:10Z)
• A Temporally Disentangled Contrastive Diffusion Model for Spatiotemporal Imputation [35.46631415365955]
  We introduce a conditional diffusion framework called C$2$TSD, which incorporates disentangled temporal (trend and seasonality) representations as conditional information. Our experiments on three real-world datasets demonstrate the superior performance of our approach compared to a number of state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-18T11:59:04Z)
• DIVERSIFY: A General Framework for Time Series Out-of-distribution Detection and Generalization [58.704753031608625]
  Time series is one of the most challenging modalities in machine learning research. OOD detection and generalization on time series tend to suffer due to its non-stationary property. We propose DIVERSIFY, a framework for OOD detection and generalization on dynamic distributions of time series.
  arXiv  Detail & Related papers  (2023-08-04T12:27:11Z)
• Leveraging the structure of dynamical systems for data-driven modeling [111.45324708884813]
  We consider the impact of the training set and its structure on the quality of the long-term prediction. We show how an informed design of the training set, based on invariants of the system and the structure of the underlying attractor, significantly improves the resulting models.
  arXiv  Detail & Related papers  (2021-12-15T20:09:20Z)
• Supporting Optimal Phase Space Reconstructions Using Neural Network Architecture for Time Series Modeling [68.8204255655161]
  We propose an artificial neural network with a mechanism to implicitly learn the phase spaces properties. Our approach is either as competitive as or better than most state-of-the-art strategies.
  arXiv  Detail & Related papers  (2020-06-19T21:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.