LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models

• URL: http://arxiv.org/abs/2608.17299v1
• Date: Tue, 18 Aug 2026 02:50:48 GMT
• Title: LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models
• Abstract summary: We introduce LiveHouse-TS, the first open-world living benchmark infrastructure for Time Series Foundation Models.<n>LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity.<n>Our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions.
• Score: 41.28223590438589
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Time Series Foundation Models (TSFMs) have recently emerged as a highly promising paradigm for cross-domain zero-shot forecasting. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how models behave in continuously evolving real-world environments characterized by seasonal variations, distribution shifts, and unexpected events. To bridge this gap, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. By evaluating models prequentially on real future data in open-world environments, LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. Rather than acting as a one-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol.

Related papers

• MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models [33.25323416656233]
  We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline.<n>Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation.<n> MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours.
  arXiv  Detail & Related papers  (2026-07-02T07:27:58Z)
• TempusBench: An Evaluation Framework for Time-Series Forecasting [36.738682337273104]
  We introduce TempusBench, an open-source evaluation framework for time-series foundation models (TSFMs)<n>We see at least four major issues impeding progress on the development of such a framework.<n>We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench.
  arXiv  Detail & Related papers  (2026-04-13T14:29:34Z)
• Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting [8.050703449557028]
  Impermanent is a live benchmark that evaluates forecasting models under open-world temporal change.<n>We focus on the top 400 repositories by star count and construct time series from issues opened, pull requests opened, push events, and new stargazers.<n>By shifting evaluation from static accuracy to sustained performance, Impermanent takes a concrete step toward assessing when and whether foundation-level generalization in time-series forecasting can be meaningfully claimed.
  arXiv  Detail & Related papers  (2026-03-09T17:59:00Z)
• MEMTS: Internalizing Domain Knowledge via Parameterized Memory for Retrieval-Free Domain Adaptation of Time Series Foundation Models [51.506429027626005]
  Memory for Time Series (MEMTS) is a lightweight and plug-and-play method for retrieval-free domain adaptation in time series forecasting.<n>Key component of MEMTS is a Knowledge Persistence Module (KPM), which internalizes domain-specific temporal dynamics.<n>This paradigm shift enables MEMTS to achieve accurate domain adaptation with constant-time inference and near-zero latency.
  arXiv  Detail & Related papers  (2026-02-14T14:00:06Z)
• It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks [87.7937890373758]
  Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation.<n>We introduce TIME, a next-generation task-centric benchmark comprising 50 fresh datasets and 98 forecasting tasks.<n>We propose a novel pattern-level evaluation perspective that moves beyond traditional dataset-level evaluations based on static meta labels.
  arXiv  Detail & Related papers  (2026-02-12T16:31:01Z)
• TS-Arena Technical Report -- A Pre-registered Live Forecasting Platform [0.0]
  Time Series Foundation Models (TSFMs) offer transformative capabilities for forecasting, but risk triggering a fundamental evaluation crisis.<n>This crisis is driven by information leakage due to overlapping training and test sets across different models, as well as the illegitimate transfer of global patterns to test data.<n>We introduce TS-Arena, a platform that restores the operational integrity of forecasting by treating the genuinely unknown future as the definitive test environment.
  arXiv  Detail & Related papers  (2025-12-23T20:48:11Z)
• Estimating Time Series Foundation Model Transferability via In-Context Learning [74.65355820906355]
  Time series foundation models (TSFMs) offer strong zero-shot forecasting via large-scale pre-training.<n>Fine-tuning remains critical for boosting performance in domains with limited public data.<n>We introduce TimeTic, a transferability estimation framework that recasts model selection as an in-context-learning problem.
  arXiv  Detail & Related papers  (2025-09-28T07:07:13Z)
• GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation [90.53485251837235]
  Time series foundation models excel in zero-shot forecasting, handling diverse tasks without explicit training. GIFT-Eval is a pioneering benchmark aimed at promoting evaluation across diverse datasets. GIFT-Eval encompasses 23 datasets over 144,000 time series and 177 million data points.
  arXiv  Detail & Related papers  (2024-10-14T11:29:38Z)
• FreDF: Learning to Forecast in the Frequency Domain [54.2091536822376]
  Time series modeling presents unique challenges due to autocorrelation in both historical data and future sequences.<n>We propose the Frequency-enhanced Direct Forecast (FreDF) which mitigates label autocorrelation by learning to forecast in the frequency domain.
  arXiv  Detail & Related papers  (2024-02-04T08:23:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.