Time Machine GPT

• URL: http://arxiv.org/abs/2404.18543v1
• Date: Mon, 29 Apr 2024 09:34:25 GMT
• Title: Time Machine GPT
• Authors: Felix Drinkall, Eghbal Rahimikia, Janet B. Pierrehumbert, Stefan Zohren,
• Abstract summary: Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora. This approach is not aligned with the evolving nature of language. This paper presents a new approach: a series of point-in-time LLMs called Time Machine GPT (TiMaGPT)
• Score: 15.661920010658626
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora, reflecting the lack of datasets with temporal metadata. This approach is not aligned with the evolving nature of language. Conventional methods for creating temporally adapted language models often depend on further pre-training static models on time-specific data. This paper presents a new approach: a series of point-in-time LLMs called Time Machine GPT (TiMaGPT), specifically designed to be nonprognosticative. This ensures they remain uninformed about future factual information and linguistic changes. This strategy is beneficial for understanding language evolution and is of critical importance when applying models in dynamic contexts, such as time-series forecasting, where foresight of future information can prove problematic. We provide access to both the models and training datasets.

Related papers

• Metadata Matters for Time Series: Informative Forecasting with Transformers [70.38241681764738]
  We propose a Metadata-informed Time Series Transformer (MetaTST) for time series forecasting. To tackle the unstructured nature of metadata, MetaTST formalizes them into natural languages by pre-designed templates. A Transformer encoder is employed to communicate series and metadata tokens, which can extend series representations by metadata information.
  arXiv  Detail & Related papers  (2024-10-04T11:37:55Z)
• AutoTimes: Autoregressive Time Series Forecasters via Large Language Models [67.83502953961505]
  AutoTimes projects time series into the embedding space of language tokens and autoregressively generates future predictions with arbitrary lengths. We formulate time series as prompts, extending the context for prediction beyond the lookback window. AutoTimes achieves state-of-the-art with 0.1% trainable parameters and over $5times$ training/inference speedup.
  arXiv  Detail & Related papers  (2024-02-04T06:59:21Z)
• Carpe Diem: On the Evaluation of World Knowledge in Lifelong Language Models [74.81091933317882]
  We introduce EvolvingQA, a temporally evolving question-answering benchmark designed for training and evaluating LMs on an evolving Wikipedia database. We uncover that existing continual learning baselines suffer from updating and removing outdated knowledge. Our work aims to model the dynamic nature of real-world information, suggesting faithful evaluations of the evolution-adaptability of language models.
  arXiv  Detail & Related papers  (2023-11-14T12:12:02Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• Lost in the Middle: How Language Models Use Long Contexts [88.78803442320246]
  We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts. We find that performance can degrade significantly when changing the position of relevant information. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.
  arXiv  Detail & Related papers  (2023-07-06T17:54:11Z)
• Jamp: Controlled Japanese Temporal Inference Dataset for Evaluating Generalization Capacity of Language Models [18.874880342410876]
  We present Jamp, a Japanese benchmark focused on temporal inference. Our dataset includes a range of temporal inference patterns, which enables us to conduct fine-grained analysis. We evaluate the generalization capacities of monolingual/multilingual LMs by splitting our dataset based on tense fragments.
  arXiv  Detail & Related papers  (2023-06-19T07:00:14Z)
• Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion [30.50032335014021]
  We propose a novel TKGC model, namely Pre-trained Language Model with Prompts for TKGC (PPT) We convert a series of sampled quadruples into pre-trained language model inputs and convert intervals between timestamps into different prompts to make coherent sentences with implicit semantic information. Our model can effectively incorporate information from temporal knowledge graphs into the language models.
  arXiv  Detail & Related papers  (2023-05-13T12:53:11Z)
• Temporal Attention for Language Models [24.34396762188068]
  We extend the key component of the transformer architecture, i.e., the self-attention mechanism, and propose temporal attention. temporal attention can be applied to any transformer model and requires the input texts to be accompanied with their relevant time points. We leverage these representations for the task of semantic change detection. Our proposed model achieves state-of-the-art results on all the datasets.
  arXiv  Detail & Related papers  (2022-02-04T11:55:34Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Time-Aware Language Models as Temporal Knowledge Bases [39.00042720454899]
  Language models (LMs) are trained on snapshots of data collected at a specific moment in time. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time. We propose a simple technique for jointly modeling text with its timestamp.
  arXiv  Detail & Related papers  (2021-06-29T06:18:57Z)
• Pitfalls of Static Language Modelling [41.76918612574081]
  We show that state-of-the-art Transformer models perform worse in the realistic setup of predicting future utterances from beyond their training period. We argue that now is the right time to rethink our static language modelling evaluation protocol.
  arXiv  Detail & Related papers  (2021-02-03T09:01:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.