Temporal Tokenization Strategies for Event Sequence Modeling with Large Language Models

• URL: http://arxiv.org/abs/2512.13618v2
• Date: Tue, 16 Dec 2025 18:35:40 GMT
• Title: Temporal Tokenization Strategies for Event Sequence Modeling with Large Language Models
• Abstract summary: This paper presents the first empirical study of temporal tokenization for event sequences.<n>We compare distinct encoding strategies: naive numeric strings, high-precision byte-level representations, human-semantic calendar tokens, classic uniform binning, and adaptive residual scalar quantization.<n>Our analysis reveals that no single strategy is universally superior; instead, prediction performance depends heavily on aligning the tokenizer with the data's statistical properties.
• Score: 8.853799051918307
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representing continuous time is a critical and under-explored challenge in modeling temporal event sequences with large language models (LLMs). Various strategies like byte-level representations or calendar tokens have been proposed. However, the optimal approach remains unclear, especially given the diverse statistical distributions of real-world event data, which range from smooth log-normal to discrete, spiky patterns. This paper presents the first empirical study of temporal tokenization for event sequences, comparing distinct encoding strategies: naive numeric strings, high-precision byte-level representations, human-semantic calendar tokens, classic uniform binning, and adaptive residual scalar quantization. We evaluate these strategies by fine-tuning LLMs on real-world datasets that exemplify these diverse distributions. Our analysis reveals that no single strategy is universally superior; instead, prediction performance depends heavily on aligning the tokenizer with the data's statistical properties, with log-based strategies excelling on skewed distributions and human-centric formats proving robust for mixed modalities.

Related papers

• Why Model Selection Fails in Time Series Forecasting: An Empirical Study of Instability Across Data Regimes [0.0]
  Time series forecasting models often exhibit inconsistent performance across datasets with varying statistical and structural properties.<n>This paper investigates why rule-based model selection fails in time series forecasting by analyzing the relationship between data-regime descriptors and model performance.
  arXiv  Detail & Related papers  (2026-05-02T21:19:29Z)
• An Enhanced Model-based Approach for Short Text Clustering [58.60681789677676]
  Short text clustering has become increasingly important with the popularity of social media like Twitter, Google+, and Facebook.<n>Existing methods can be broadly categorized into two paradigms: topic model-based approaches and deep representation learning-based approaches.<n>We propose a collapsed Gibbs Sampling algorithm for the Dirichlet Multinomial Mixture model (GSDMM), which effectively handles the sparsity and high dimensionality of short texts.<n>Based on several aspects of GSDMM that warrant further refinement, we propose an improved approach, GSDMM+, designed to further optimize its performance.
  arXiv  Detail & Related papers  (2025-07-18T10:07:42Z)
• BLAST: Balanced Sampling Time Series Corpus for Universal Forecasting Models [47.66064662912721]
  We introduce a novel pre-training corpus designed to enhance data diversity through a balanced sampling strategy.<n>BLT incorporates 321 billion observations from publicly available datasets and employs a comprehensive suite of statistical metrics to characterize time series patterns.<n>Our findings highlight the pivotal role of data diversity in improving both training efficiency and model performance for the universal forecasting task.
  arXiv  Detail & Related papers  (2025-05-23T13:20:47Z)
• Bridging Distribution Gaps in Time Series Foundation Model Pretraining with Prototype-Guided Normalization [29.082583523943157]
  We propose a domain-aware adaptive normalization strategy within the Transformer architecture.<n>We replace the traditional LayerNorm with a prototype-guided dynamic normalization mechanism (ProtoNorm)<n>Our method significantly outperforms conventional pretraining techniques across both classification and forecasting tasks.
  arXiv  Detail & Related papers  (2025-04-15T06:23:00Z)
• Embedded feature selection in LSTM networks with multi-objective evolutionary ensemble learning for time series forecasting [49.1574468325115]
  We present a novel feature selection method embedded in Long Short-Term Memory networks. Our approach optimize the weights and biases of the LSTM in a partitioned manner. Experimental evaluations on air quality time series data from Italy and southeast Spain demonstrate that our method substantially improves the ability generalization of conventional LSTMs.
  arXiv  Detail & Related papers  (2023-12-29T08:42:10Z)
• It's All in the Mix: Wasserstein Classification and Regression with Mixed Features [2.2685251390114565]
  We develop and analyze distributionally robust prediction models that faithfully account for the presence of discrete features.<n>We demonstrate that our models can significantly outperform existing methods that are agnostic to the presence of discrete features.
  arXiv  Detail & Related papers  (2023-12-19T15:15:52Z)
• Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
  Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers. We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes. We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-28T18:48:34Z)
• Large-scale Fully-Unsupervised Re-Identification [78.47108158030213]
  We propose two strategies to learn from large-scale unlabeled data. The first strategy performs a local neighborhood sampling to reduce the dataset size in each without violating neighborhood relationships. A second strategy leverages a novel Re-Ranking technique, which has a lower time upper bound complexity and reduces the memory complexity from O(n2) to O(kn) with k n.
  arXiv  Detail & Related papers  (2023-07-26T16:19:19Z)
• Continuous-time convolutions model of event sequences [46.3471121117337]
  Event sequences are non-uniform and sparse, making traditional models unsuitable. We propose COTIC, a method based on an efficient convolution neural network designed to handle the non-uniform occurrence of events over time. COTIC outperforms existing models in predicting the next event time and type, achieving an average rank of 1.5 compared to 3.714 for the nearest competitor.
  arXiv  Detail & Related papers  (2023-02-13T10:34:51Z)
• Hierarchical regularization networks for sparsification based learning on noisy datasets [0.0]
  hierarchy follows from approximation spaces identified at successively finer scales. For promoting model generalization at each scale, we also introduce a novel, projection based penalty operator across multiple dimension. Results show the performance of the approach as a data reduction and modeling strategy on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2020-06-09T18:32:24Z)
• A Multi-Channel Neural Graphical Event Model with Negative Evidence [76.51278722190607]
  Event datasets are sequences of events of various types occurring irregularly over the time-line. We propose a non-parametric deep neural network approach in order to estimate the underlying intensity functions.
  arXiv  Detail & Related papers  (2020-02-21T23:10:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.