TempusBench: An Evaluation Framework for Time-Series Forecasting

• URL: http://arxiv.org/abs/2604.11529v2
• Date: Thu, 16 Apr 2026 16:57:53 GMT
• Title: TempusBench: An Evaluation Framework for Time-Series Forecasting
• Abstract summary: We introduce TempusBench, an open-source evaluation framework for time-series foundation models (TSFMs)<n>We see at least four major issues impeding progress on the development of such a framework.<n>We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench.
• Score: 36.738682337273104
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://benchmark.smlcrm.com/.

Related papers

• Beyond Numerical Time Series: A Unified Benchmark for Multimodal Forecasting with Heterogeneous Context [10.330435945681916]
  MUSE-Bench is a unified benchmark for multimodal time series forecasting with heterogeneous context.<n>It comprises fourteen datasets across eight domains and six types of context.
  arXiv  Detail & Related papers  (2026-09-14T06:01:16Z)
• LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models [41.28223590438589]
  We introduce LiveHouse-TS, the first open-world living benchmark infrastructure for Time Series Foundation Models.<n>LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity.<n>Our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions.
  arXiv  Detail & Related papers  (2026-08-18T02:50:48Z)
• MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents [50.27505102982638]
  We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents.<n>The framework provides a stateful execution environment spanning 500+ tools across 16 application domains.<n>It supports multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls.
  arXiv  Detail & Related papers  (2026-07-13T17:13:09Z)
• It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks [87.7937890373758]
  Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation.<n>We introduce TIME, a next-generation task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks.<n>We propose a novel pattern-level evaluation perspective that moves beyond traditional dataset-level evaluations based on static meta labels.
  arXiv  Detail & Related papers  (2026-02-12T16:31:01Z)
• Constantly Improving Image Models Need Constantly Improving Benchmarks [109.39018167487103]
  We present ECHO, a framework for constructing benchmarks directly from real-world evidence of model use.<n>Applying this framework to GPT-4o Image Gen, we construct a dataset of over 31,000 prompts curated from social media posts.
  arXiv  Detail & Related papers  (2025-10-16T17:59:30Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence [61.46575527504109]
  LimiX-16M and LimiX-2M treat structured data as a joint distribution over variables and missingness.<n>We evaluate LimiX models across 11 large structured-data benchmarks with broad regimes of sample size, feature dimensionality, class number, categorical-to-numerical feature ratio, missingness, and sample-to-feature ratios.
  arXiv  Detail & Related papers  (2025-09-03T17:39:08Z)
• Multi-Scale Finetuning for Encoder-based Time Series Foundation Models [67.95907033226585]
  Time series foundation models (TSFMs) demonstrate impressive zero-shot performance for time series forecasting.<n>While naive finetuning can yield performance gains, we argue that it falls short of fully leveraging TSFMs' capabilities.<n>We propose Multiscale finetuning (MSFT), a simple yet general framework that explicitly integrates multi-scale modeling into the finetuning process.
  arXiv  Detail & Related papers  (2025-06-17T01:06:01Z)
• TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting [35.505530132151]
  Time Series Forecasting (TSF) is key functionality in numerous fields, such as financial investment, weather services, and energy management.<n>Many TSF methods require domain-specific data collection and model training and do not generalize well when applied in other domains.<n>Time Series Foundation Models (TSFMs) that are pre-trained on massive heterogeneous time series data aim to overcome these limitations.
  arXiv  Detail & Related papers  (2024-10-15T17:23:49Z)
• The Bayesian Context Trees State Space Model for time series modelling and forecasting [7.018547803286913]
  A hierarchical Bayesian framework is introduced for developing tree-based mixture models for time series.<n>We call this the Bayesian Context Trees State Space Model, or the BCT-X framework.
  arXiv  Detail & Related papers  (2023-08-02T02:40:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.