Understanding the Limits of Deep Tabular Methods with Temporal Shift

• URL: http://arxiv.org/abs/2502.20260v1
• Date: Thu, 27 Feb 2025 16:48:53 GMT
• Title: Understanding the Limits of Deep Tabular Methods with Temporal Shift
• Authors: Hao-Run Cai, Han-Jia Ye,
• Abstract summary: We introduce a plug-and-play temporal embedding method based on Fourier series expansion to learn and incorporate temporal patterns.<n>Our experiments demonstrate that this temporal embedding, combined with the improved training protocol, provides a more effective and robust framework for learning from temporal data.
• Score: 28.738848567072004
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deep tabular models have demonstrated remarkable success on i.i.d. data, excelling in a variety of structured data tasks. However, their performance often deteriorates under temporal distribution shifts, where trends and periodic patterns are present in the evolving data distribution over time. In this paper, we explore the underlying reasons for this failure in capturing temporal dependencies. We begin by investigating the training protocol, revealing a key issue in how model selection perform. While existing approaches use temporal ordering for splitting validation set, we show that even a random split can significantly improve model performance. By minimizing the time lag between training data and test time, while reducing the bias in validation, our proposed training protocol significantly improves generalization across various methods. Furthermore, we analyze how temporal data affects deep tabular representations, uncovering that these models often fail to capture crucial periodic and trend information. To address this gap, we introduce a plug-and-play temporal embedding method based on Fourier series expansion to learn and incorporate temporal patterns, offering an adaptive approach to handle temporal shifts. Our experiments demonstrate that this temporal embedding, combined with the improved training protocol, provides a more effective and robust framework for learning from temporal tabular data.

Related papers

• PSDNorm: Test-Time Temporal Normalization for Deep Learning on EEG Signals [63.05435596565677]
  PSDNorm is a layer that leverages Monge mapping and temporal context to normalize feature maps in deep learning models. PSDNorm achieves state-of-the-art performance at test time on datasets not seen during training. PSDNorm provides a significant improvement in robustness, achieving markedly higher F1 scores for the 20% hardest subjects.
  arXiv  Detail & Related papers  (2025-03-06T16:20:25Z)
• Capturing the Temporal Dependence of Training Data Influence [100.91355498124527]
  We formalize the concept of trajectory-specific leave-one-out influence, which quantifies the impact of removing a data point during training. We propose data value embedding, a novel technique enabling efficient approximation of trajectory-specific LOO. As data value embedding captures training data ordering, it offers valuable insights into model training dynamics.
  arXiv  Detail & Related papers  (2024-12-12T18:28:55Z)
• Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization [74.3339999119713]
  We develop a wavelet-based tokenizer that allows models to learn complex representations directly in the space of time-localized frequencies. Our method first scales and decomposes the input time series, then thresholds and quantizes the wavelet coefficients, and finally pre-trains an autoregressive model to forecast coefficients for the forecast horizon.
  arXiv  Detail & Related papers  (2024-12-06T18:22:59Z)
• On the Regularization of Learnable Embeddings for Time Series Forecasting [18.069747511100132]
  We investigate methods to regularize the learning of local learnable embeddings for time series processing. We show that methods attempting to prevent the co-adaptation of local and global parameters by means of embeddings perturbation are particularly effective in this context.
  arXiv  Detail & Related papers  (2024-10-18T17:30:20Z)
• TimeInf: Time Series Data Contribution via Influence Functions [8.018453062120916]
  TimeInf is a data contribution estimation method for time-series datasets. Our empirical results demonstrate that TimeInf outperforms state-of-the-art methods in identifying harmful anomalies. TimeInf offers intuitive and interpretable attributions of data values, allowing us to easily distinguish diverse anomaly patterns through visualizations.
  arXiv  Detail & Related papers  (2024-07-21T19:10:40Z)
• tPARAFAC2: Tracking evolving patterns in (incomplete) temporal data [0.7285444492473742]
  We introduce t(emporal)PARAFAC2 which utilizes temporal smoothness regularization on the evolving factors. Our numerical experiments on both simulated and real datasets demonstrate the effectiveness of the temporal smoothness regularization.
  arXiv  Detail & Related papers  (2024-07-01T15:10:55Z)
• DeformTime: Capturing Variable Dependencies with Deformable Attention for Time Series Forecasting [0.34530027457862006]
  We present DeformTime, a neural network architecture that attempts to capture correlated temporal patterns from the input space. We conduct extensive experiments on 6 MTS data sets, using previously established benchmarks as well as challenging infectious disease modelling tasks. Results demonstrate that DeformTime improves accuracy against previous competitive methods across the vast majority of MTS forecasting tasks.
  arXiv  Detail & Related papers  (2024-06-11T16:45:48Z)
• ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks [1.3723120574076126]
  This study investigates the challenges posed by the dynamic nature of legal multi-label text classification tasks, where legal concepts evolve over time. Existing models often overlook the temporal dimension in their training process, leading to suboptimal performance of those models over time. We introduce ChronosLex, an incremental training paradigm that trains models on chronological splits, preserving the temporal order of the data.
  arXiv  Detail & Related papers  (2024-05-23T06:09:16Z)
• TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling [67.02157180089573]
  Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. This paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks.
  arXiv  Detail & Related papers  (2024-02-04T13:10:51Z)
• Data Attribution for Diffusion Models: Timestep-induced Bias in Influence Estimation [53.27596811146316]
  Diffusion models operate over a sequence of timesteps instead of instantaneous input-output relationships in previous contexts. We present Diffusion-TracIn that incorporates this temporal dynamics and observe that samples' loss gradient norms are highly dependent on timestep. We introduce Diffusion-ReTrac as a re-normalized adaptation that enables the retrieval of training samples more targeted to the test sample of interest.
  arXiv  Detail & Related papers  (2024-01-17T07:58:18Z)
• Time-Varying Propensity Score to Bridge the Gap between the Past and Present [104.46387765330142]
  We introduce a time-varying propensity score that can detect gradual shifts in the distribution of data. We demonstrate different ways of implementing it and evaluate it on a variety of problems.
  arXiv  Detail & Related papers  (2022-10-04T07:21:49Z)
• Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time [26.261277497201565]
  In several real world applications, machine learning models are deployed to make predictions on data whose distribution changes gradually along time. We propose a simple method that starts with a model with time-sensitive parameters but regularizes its temporal complexity using a Gradient Interpolation (GI) loss.
  arXiv  Detail & Related papers  (2021-08-15T11:20:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.