Large Language Models are Few-shot Multivariate Time Series Classifiers

• URL: http://arxiv.org/abs/2502.00059v1
• Date: Thu, 30 Jan 2025 03:59:59 GMT
• Title: Large Language Models are Few-shot Multivariate Time Series Classifiers
• Authors: Yakun Chen, Zihao Li, Chao Yang, Xianzhi Wang, Guandong Xu,
• Abstract summary: Large Language Models (LLMs) have been extensively applied in time series analysis. Yet, their utility in the few-shot classification (i.e., a crucial training scenario) is underexplored. We aim to leverage the extensive pre-trained knowledge in LLMs to overcome the data scarcity problem.
• Score: 23.045734479292356
• License:
• Abstract: Large Language Models (LLMs) have been extensively applied in time series analysis. Yet, their utility in the few-shot classification (i.e., a crucial training scenario due to the limited training data available in industrial applications) concerning multivariate time series data remains underexplored. We aim to leverage the extensive pre-trained knowledge in LLMs to overcome the data scarcity problem within multivariate time series. Specifically, we propose LLMFew, an LLM-enhanced framework to investigate the feasibility and capacity of LLMs for few-shot multivariate time series classification. This model introduces a Patch-wise Temporal Convolution Encoder (PTCEnc) to align time series data with the textual embedding input of LLMs. We further fine-tune the pre-trained LLM decoder with Low-rank Adaptations (LoRA) to enhance its feature representation learning ability in time series data. Experimental results show that our model outperformed state-of-the-art baselines by a large margin, achieving 125.2% and 50.2% improvement in classification accuracy on Handwriting and EthanolConcentration datasets, respectively. Moreover, our experimental results demonstrate that LLM-based methods perform well across a variety of datasets in few-shot MTSC, delivering reliable results compared to traditional models. This success paves the way for their deployment in industrial environments where data are limited.

Related papers

• Using Pre-trained LLMs for Multivariate Time Series Forecasting [41.67881279885103]
  Pre-trained Large Language Models (LLMs) encapsulate large amounts of knowledge and take enormous amounts of compute to train. We make use of this resource, together with the observation that LLMs are able to transfer knowledge and performance from one domain or even modality to another seemingly-unrelated area.
  arXiv  Detail & Related papers  (2025-01-10T23:30:23Z)
• M-CELS: Counterfactual Explanation for Multivariate Time Series Data Guided by Learned Saliency Maps [0.9374652839580181]
  We introduce M-CELS, a counterfactual explanation model designed to enhance interpretability in multidimensional time series classification tasks. Results demonstrate the superior performance of M-CELS in terms of validity, proximity, and sparsity.
  arXiv  Detail & Related papers  (2024-11-04T22:16:24Z)
• Revisited Large Language Model for Time Series Analysis through Modality Alignment [16.147350486106777]
  Large Language Models have demonstrated impressive performance in many pivotal web applications such as sensor data analysis. In this study, we assess the effectiveness of applying LLMs to key time series tasks, including forecasting, classification, imputation, and anomaly detection. Our results reveal that LLMs offer minimal advantages for these core time series tasks and may even distort the temporal structure of the data.
  arXiv  Detail & Related papers  (2024-10-16T07:47:31Z)
• 60 Data Points are Sufficient to Fine-Tune LLMs for Question-Answering [50.12622877002846]
  Large language models (LLMs) encode extensive world knowledge through pre-training on massive datasets, which can be fine-tuned for the question-answering (QA) task. We categorize supervised fine-tuning (SFT) data based on the extent of knowledge memorized by the pretrained LLMs. Our experiments show that as few as 60 data points during the SFT stage can activate the knowledge encoded during pre-training, enabling LLMs to perform the QA task.
  arXiv  Detail & Related papers  (2024-09-24T07:38:38Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilities [46.02234423159257]
  Large language models (LLMs) have been applied in many fields and have developed rapidly in recent years. Recent works treat large language models as emphzero-shot time series reasoners without further fine-tuning. Our study shows that LLMs perform well in predicting time series with clear patterns and trends, but face challenges with datasets lacking periodicity.
  arXiv  Detail & Related papers  (2024-02-16T17:15:28Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters [11.796765525301051]
  We propose a framework for time-series forecasting with pre-trained Large Language Models (LLMs) LLM4TS consists of a two-stage fine-tuning strategy to align LLMs with the nuances of time-series data, and the forecasting fine-tuning stage for downstream time-series forecasting tasks. Our framework features a novel two-level aggregation method that integrates multi-scale temporal data within pre-trained LLMs, enhancing their ability to interpret time-specific information.
  arXiv  Detail & Related papers  (2023-08-16T16:19:50Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• Learning summary features of time series for likelihood free inference [93.08098361687722]
  We present a data-driven strategy for automatically learning summary features from time series data. Our results indicate that learning summary features from data can compete and even outperform LFI methods based on hand-crafted values.
  arXiv  Detail & Related papers  (2020-12-04T19:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.