TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale

• URL: http://arxiv.org/abs/2604.10291v1
• Date: Sat, 11 Apr 2026 17:15:26 GMT
• Title: TimeSeriesExamAgent: Creating Time Series Reasoning Benchmarks at Scale
• Abstract summary: Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data?<n>We propose scalable methods for creating comprehensive time series reasoning benchmarks that combine the flexibility of templates with the creativity of LLM agents.<n>Our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specific applications.
• Score: 19.848029471559624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have shown promising performance in time series modeling tasks, but do they truly understand time series data? While multiple benchmarks have been proposed to answer this fundamental question, most are manually curated and focus on narrow domains or specific skill sets. To address this limitation, we propose scalable methods for creating comprehensive time series reasoning benchmarks that combine the flexibility of templates with the creativity of LLM agents. We first develop TimeSeriesExam, a multiple-choice benchmark using synthetic time series to evaluate LLMs across five core reasoning categories: pattern recognitionnoise understandingsimilarity analysisanomaly detection, and causality. Then, with TimeSeriesExamAgent, we scale our approach by automatically generating benchmarks from real-world datasets spanning healthcare, finance and weather domains. Through multi-dimensional quality evaluation, we demonstrate that our automatically generated benchmarks achieve diversity comparable to manually curated alternatives. However, our experiments reveal that LLM performance remains limited in both abstract time series reasoning and domain-specific applications, highlighting ongoing challenges in enabling effective time series understanding in these models. TimeSeriesExamAgent is available at https://github.com/magwiazda/TimeSeriesExamAgent.

Related papers

• TSAQA: Time Series Analysis Question And Answering Benchmark [85.35545785252309]
  Time series data are integral to critical applications across domains such as finance, healthcare, transportation, and environmental science.<n>We introduce TSAQA, a novel unified benchmark designed to broaden task coverage and evaluate diverse temporal analysis capabilities.
  arXiv  Detail & Related papers  (2026-01-30T17:28:56Z)
• Time-RA: Towards Time Series Reasoning for Anomaly with LLM Feedback [55.284574165467525]
  Time-series Reasoning for Anomaly (Time-RA) transforms classical time series anomaly detection into a generative, reasoning-intensive task.<n>Also, we introduce the first real-world multimodal benchmark dataset, RATs40K, explicitly annotated for anomaly reasoning.
  arXiv  Detail & Related papers  (2025-07-20T18:02:50Z)
• Towards Cross-Modality Modeling for Time Series Analytics: A Survey in the LLM Era [24.980206999214552]
  Large Language Models (LLMs) have emerged as a new paradigm for time series analytics.<n>LLMs are pre-trained on textual corpora and are not inherently optimized for time series.<n>This survey is designed for a range of professionals, researchers, and practitioners interested in LLM-based time series modeling.
  arXiv  Detail & Related papers  (2025-05-05T11:35:33Z)
• LLM-PS: Empowering Large Language Models for Time Series Forecasting with Temporal Patterns and Semantics [56.99021951927683]
  Time Series Forecasting (TSF) is critical in many real-world domains like financial planning and health monitoring.<n>Existing Large Language Models (LLMs) usually perform suboptimally because they neglect the inherent characteristics of time series data.<n>We propose LLM-PS to empower the LLM for TSF by learning the fundamental textitPatterns and meaningful textitSemantics from time series data.
  arXiv  Detail & Related papers  (2025-03-12T11:45:11Z)
• LAST SToP For Modeling Asynchronous Time Series [19.401463051705377]
  We present a novel prompt design for Large Language Models (LLMs) tailored to Asynchronous Time Series.<n>Our approach effectively utilizes the rich natural language of event descriptions, allowing LLMs to benefit from their broad world knowledge for reasoning across different domains and tasks.<n>We further introduce Soft Prompting, a novel prompt-tuning mechanism that significantly improves model performance, outperforming existing fine-tuning methods such as QLoRA.
  arXiv  Detail & Related papers  (2025-02-04T01:42:45Z)
• ChatTS: Aligning Time Series with LLMs via Synthetic Data for Enhanced Understanding and Reasoning [10.854285913078257]
  This paper introduces ChatTS, a novel MLLM designed for time series analysis.<n>ChatTS treats time series as a modality, similar to how vision MLLMs process images.<n>Time Series Evol-Instruct generates diverse time series Q&As, enhancing the model's reasoning capabilities.
  arXiv  Detail & Related papers  (2024-12-04T08:06:15Z)
• Hierarchical Multimodal LLMs with Semantic Space Alignment for Enhanced Time Series Classification [4.5939667818289385]
  HiTime is a hierarchical multi-modal model that seamlessly integrates temporal information into large language models. Our findings highlight the potential of integrating temporal features into LLMs, paving the way for advanced time series analysis.
  arXiv  Detail & Related papers  (2024-10-24T12:32:19Z)
• Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts [103.725112190618]
  This paper introduces Moirai-MoE, using a single input/output projection layer while delegating the modeling of diverse time series patterns to the sparse mixture of experts. Extensive experiments on 39 datasets demonstrate the superiority of Moirai-MoE over existing foundation models in both in-distribution and zero-shot scenarios.
  arXiv  Detail & Related papers  (2024-10-14T13:01:11Z)
• Deep Time Series Models: A Comprehensive Survey and Benchmark [60.742416934632416]
  Time series present unique challenges due to their intricate and dynamic nature.<n>Recent years have witnessed remarkable breakthroughs in the time series community.<n>We release Time Series Library (TSLib) as a fair benchmark of deep time series models for diverse analysis tasks.
  arXiv  Detail & Related papers  (2024-07-18T08:31:55Z)
• TimeLDM: Latent Diffusion Model for Unconditional Time Series Generation [2.4454605633840143]
  Time series generation is a crucial research topic in the area of decision-making systems. Recent approaches focus on learning in the data space to model time series information. We propose TimeLDM, a novel latent diffusion model for high-quality time series generation.
  arXiv  Detail & Related papers  (2024-07-05T01:47:20Z)
• AutoTimes: Autoregressive Time Series Forecasters via Large Language Models [67.83502953961505]
  AutoTimes projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. We formulate time series as prompts, extending the context for prediction beyond the lookback window. AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5times$ training/inference speedup.
  arXiv  Detail & Related papers  (2024-02-04T06:59:21Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.