Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Series

• URL: http://arxiv.org/abs/2603.06605v2
• Date: Thu, 12 Mar 2026 05:39:02 GMT
• Title: Structure-Aware Set Transformers: Temporal and Variable-Type Attention Biases for Asynchronous Clinical Time Series
• Abstract summary: We show how to restore priors in STructure-AwaRe (STAR) Set Transformer.<n>On three ICU prediction tasks, STAR-Set achieves AUC/APR of 0.7158/0.0026 (CPR), 0.9164/0.2033 (mortality), and 0.8373/0.1258 (vasopressor use)
• Score: 38.229225431387015
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Electronic health records (EHR) are irregular, asynchronous multivariate time series. As time-series foundation models increasingly tokenize events rather than discretizing time, the input layout becomes a key design choice. Grids expose time$\times$variable structure but require imputation or missingness masks, risking error or sampling-policy shortcuts. Point-set tokenization avoids discretization but loses within-variable trajectories and time-local cross-variable context (Fig.1). We restore these priors in STructure-AwaRe (STAR) Set Transformer by adding parameter-efficient soft attention biases: a temporal locality penalty $-|Δt|/τ$ with learnable timescales and a variable-type affinity $B_{s_i,s_j}$ from a learned feature-compatibility matrix. We benchmark 10 depth-wise fusion schedules (Fig.2). On three ICU prediction tasks, STAR-Set achieves AUC/APR of 0.7158/0.0026 (CPR), 0.9164/0.2033 (mortality), and 0.8373/0.1258 (vasopressor use), outperforming regular-grid, event-time grid, and prior set baselines. Learned $τ$ and $B$ provide interpretable summaries of temporal context and variable interactions, offering a practical plug-in for context-informed time-series models.

Related papers

• TIE: Time Interval Encoding for Video Generation over Events [50.66585165263848]
  Director-style prompting, robotic action prediction, and interactive video agents demand temporal grounding over concurrent events.<n>Modern video generators represent time as discrete points through point-wise positional encodings.<n>Time Interval TIE is a principled, plug-and-play interval-aware generalization.
  arXiv  Detail & Related papers  (2026-05-11T13:23:14Z)
• Time-Aware Prior Fitted Networks for Zero-Shot Forecasting with Exogenous Variables [59.37476887007663]
  We develop ApolloPFN, a prior-data fitted network (PFN) that is time-aware (unlike prior PFNs)<n>Our design introduces two major advances: (i) a synthetic data generation procedure tailored to resolve the failure modes that arise when PFNs are applied to time series; and (ii) time-aware architectural modifications that embed inductive biases needed to exploit the time series context.
  arXiv  Detail & Related papers  (2026-03-16T18:33:23Z)
• T1: One-to-One Channel-Head Binding for Multivariate Time-Series Imputation [6.002503434201551]
  Imputing missing values in time series is challenging under diverse missing patterns and heavy missingness.<n>We introduce T1 (Time series imputation with 1-to-1 channel-head binding), a CNN-Transformer hybrid architecture that achieves robust imputation.<n>T1 achieves state-of-the-art performance, reducing MSE by 46% on average compared to the second-best baseline.
  arXiv  Detail & Related papers  (2026-02-24T16:03:46Z)
• PiXTime: A Model for Federated Time Series Forecasting with Heterogeneous Data Structures Across Nodes [52.821072802825654]
  Time series are highly valuable and rarely shareable across nodes.<n>Different sampling standards lead to diverse time granularities and variable sets across nodes.<n>We propose PiXTime, a novel time series forecasting model designed for federated learning.
  arXiv  Detail & Related papers  (2026-01-09T08:11:45Z)
• Kairos: Towards Adaptive and Generalizable Time Series Foundation Models [27.076542021368056]
  Time series foundation models (TSFMs) have emerged as a powerful paradigm for time series analysis.<n>We propose Kairos, a flexible TSFM framework that integrates a dynamic patching tokenizer and an instance-adaptive positional embedding.<n>Kairos achieves superior performance with much fewer parameters on two common zero-shot benchmarks.
  arXiv  Detail & Related papers  (2025-09-30T06:02:26Z)
• Timer-XL: Long-Context Transformers for Unified Time Series Forecasting [67.83502953961505]
  We present Timer-XL, a causal Transformer for unified time series forecasting.<n>Based on large-scale pre-training, Timer-XL achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-10-07T07:27:39Z)
• Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers [55.475142494272724]
  Time series prediction is crucial for understanding and forecasting complex dynamics in various domains. We introduce GridTST, a model that combines the benefits of two approaches using innovative multi-directional attentions. The model consistently delivers state-of-the-art performance across various real-world datasets.
  arXiv  Detail & Related papers  (2024-05-22T16:41:21Z)
• Long-Range Transformers for Dynamic Spatiotemporal Forecasting [16.37467119526305]
  Methods based on graph neural networks explicitly model variable relationships. Long-Range Transformers can learn interactions between time, value, and information jointly along this extended sequence.
  arXiv  Detail & Related papers  (2021-09-24T22:11:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.