Fine-grained Attention in Hierarchical Transformers for Tabular Time-series

• URL: http://arxiv.org/abs/2406.15327v2
• Date: Fri, 2 Aug 2024 13:25:16 GMT
• Title: Fine-grained Attention in Hierarchical Transformers for Tabular Time-series
• Abstract summary: We propose Fieldy, a fine-grained hierarchical model that contextualizes fields at both the row and column levels. Our results show that combining row-wise and column-wise attention improves performance without increasing model size.
• Score: 46.96700822457507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tabular data is ubiquitous in many real-life systems. In particular, time-dependent tabular data, where rows are chronologically related, is typically used for recording historical events, e.g., financial transactions, healthcare records, or stock history. Recently, hierarchical variants of the attention mechanism of transformer architectures have been used to model tabular time-series data. At first, rows (or columns) are encoded separately by computing attention between their fields. Subsequently, encoded rows (or columns) are attended to one another to model the entire tabular time-series. While efficient, this approach constrains the attention granularity and limits its ability to learn patterns at the field-level across separate rows, or columns. We take a first step to address this gap by proposing Fieldy, a fine-grained hierarchical model that contextualizes fields at both the row and column levels. We compare our proposal against state of the art models on regression and classification tasks using public tabular time-series datasets. Our results show that combining row-wise and column-wise attention improves performance without increasing model size. Code and data are available at https://github.com/raphaaal/fieldy.

Related papers

• Discrete Prototypical Memories for Federated Time Series Foundation Models [62.22804699052539]
  Leveraging Large Language Models as federated learning (FL)-based time series foundation models offers a promising way to transfer the generalization capabilities of LLMs to time series data.<n>However, the semantic misalignment between time-series data and the text-centric latent space of existing LLMs often leads to degraded performance.<n>We propose textscFeDPM, a federated framework for time-series foundation models based on discrete prototypical memories.
  arXiv  Detail & Related papers  (2026-04-06T06:57:08Z)
• PiXTime: A Model for Federated Time Series Forecasting with Heterogeneous Data Structures Across Nodes [52.821072802825654]
  Time series are highly valuable and rarely shareable across nodes.<n>Different sampling standards lead to diverse time granularities and variable sets across nodes.<n>We propose PiXTime, a novel time series forecasting model designed for federated learning.
  arXiv  Detail & Related papers  (2026-01-09T08:11:45Z)
• MemoriesDB: A Temporal-Semantic-Relational Database for Long-Term Agent Memory / Modeling Experience as a Graph of Temporal-Semantic Surfaces [0.0]
  We introduce MemoriesDB, a unified data architecture designed to avoid decoherence across time, meaning, and relation.<n>Each memory is a time-semantic-relational entity that simultaneously encodes when an event occurred, what it means, and how it connects to other events.
  arXiv  Detail & Related papers  (2025-11-09T01:41:55Z)
• TabDiff: a Multi-Modal Diffusion Model for Tabular Data Generation [91.50296404732902]
  We introduce TabDiff, a joint diffusion framework that models all multi-modal distributions of tabular data in one model. Our key innovation is the development of a joint continuous-time diffusion process for numerical and categorical data. TabDiff achieves superior average performance over existing competitive baselines, with up to $22.5%$ improvement over the state-of-the-art model on pair-wise column correlation estimations.
  arXiv  Detail & Related papers  (2024-10-27T22:58:47Z)
• Timeseria: an object-oriented time series processing library [0.40964539027092917]
  Timeseria is an object-oriented time series processing library implemented in Python. It aims at making it easier to manipulate time series data and to build statistical and machine learning models on top of it.
  arXiv  Detail & Related papers  (2024-10-12T15:29:18Z)
• Table Transformers for Imputing Textual Attributes [15.823533688884105]
  We propose a novel end-to-end approach called Table Transformers for Imputing Textual Attributes (TTITA) Our approach shows competitive performance outperforming baseline models such as recurrent neural networks and Llama2. We incorporate multi-task learning to simultaneously impute for heterogeneous columns, boosting the performance for text imputation.
  arXiv  Detail & Related papers  (2024-08-04T19:54:12Z)
• Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers [55.475142494272724]
  Time series prediction is crucial for understanding and forecasting complex dynamics in various domains. We introduce GridTST, a model that combines the benefits of two approaches using innovative multi-directional attentions. The model consistently delivers state-of-the-art performance across various real-world datasets.
  arXiv  Detail & Related papers  (2024-05-22T16:41:21Z)
• Efficient High-Resolution Time Series Classification via Attention Kronecker Decomposition [17.71968215237596]
  High-resolution time series classification is essential due to the increasing availability of detailed temporal data in various domains. We propose a new time series transformer backbone (KronTime) by introducing Kronecker-decomposed attention to process such multi-level time series. Experiments on four long time series datasets demonstrate superior classification results with improved efficiency compared to baseline methods.
  arXiv  Detail & Related papers  (2024-03-07T20:14:20Z)
• TimeSiam: A Pre-Training Framework for Siamese Time-Series Modeling [67.02157180089573]
  Time series pre-training has recently garnered wide attention for its potential to reduce labeling expenses and benefit various downstream tasks. This paper proposes TimeSiam as a simple but effective self-supervised pre-training framework for Time series based on Siamese networks.
  arXiv  Detail & Related papers  (2024-02-04T13:10:51Z)
• Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domain [54.67888148566323]
  We introduce three large-scale time series forecasting datasets from the cloud operations domain. We show it is a strong zero-shot baseline and benefits from further scaling, both in model and dataset size. Accompanying these datasets and results is a suite of comprehensive benchmark results comparing classical and deep learning baselines to our pre-trained method.
  arXiv  Detail & Related papers  (2023-10-08T08:09:51Z)
• TIGER: Temporal Interaction Graph Embedding with Restarts [12.685645074210562]
  Temporal interaction graphs (TIGs) are prevalent in fields like e-commerce and social networks. TIGs consist of sequences of timestamped interaction events that vary over time. Previous methods have to process the sequence of events chronologically and consecutively to ensure node representations are up-to-date. This prevents existing models from parallelization and reduces their flexibility in industrial applications.
  arXiv  Detail & Related papers  (2023-02-13T02:29:11Z)
• Attention Augmented Convolutional Transformer for Tabular Time-series [0.9137554315375922]
  Time-series classification is one of the most frequently performed tasks in industrial data science. We propose a novel scalable architecture for learning representations from time-series data. Our proposed model is end-to-end and can handle both categorical and continuous valued inputs.
  arXiv  Detail & Related papers  (2021-10-05T05:20:46Z)
• IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation [61.09660709356527]
  We propose a database schema interaction graph encoder to utilize historicalal information of database schema items. We evaluate our model on the benchmark SParC and Co datasets.
  arXiv  Detail & Related papers  (2020-11-11T12:56:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.