Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events

• URL: http://arxiv.org/abs/2306.11547v2
• Date: Wed, 21 Jun 2023 14:02:02 GMT
• Title: Event Stream GPT: A Data Pre-processing and Modeling Library for Generative, Pre-trained Transformers over Continuous-time Sequences of Complex Events
• Authors: Matthew B. A. McDermott, Bret Nestor, Peniel Argaw, Isaac Kohane
• Abstract summary: Event Stream GPT (ESGPT) is an open-source library designed to streamline the end-to-end process for building GPTs for continuous-time event sequences. ESGPT allows users to build flexible, foundation-model scale input datasets by specifying only a minimal configuration file.
• Score: 2.9330609943398525
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Generative, pre-trained transformers (GPTs, a.k.a. "Foundation Models") have reshaped natural language processing (NLP) through their versatility in diverse downstream tasks. However, their potential extends far beyond NLP. This paper provides a software utility to help realize this potential, extending the applicability of GPTs to continuous-time sequences of complex events with internal dependencies, such as medical record datasets. Despite their potential, the adoption of foundation models in these domains has been hampered by the lack of suitable tools for model construction and evaluation. To bridge this gap, we introduce Event Stream GPT (ESGPT), an open-source library designed to streamline the end-to-end process for building GPTs for continuous-time event sequences. ESGPT allows users to (1) build flexible, foundation-model scale input datasets by specifying only a minimal configuration file, (2) leverage a Hugging Face compatible modeling API for GPTs over this modality that incorporates intra-event causal dependency structures and autoregressive generation capabilities, and (3) evaluate models via standardized processes that can assess few and even zero-shot performance of pre-trained models on user-specified fine-tuning tasks.

Related papers

• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• Generative Pretrained Hierarchical Transformer for Time Series Forecasting [3.739587363053192]
  We propose a novel generative pretrained hierarchical transformer architecture for forecasting, named textbfGPHT. We conduct sufficient experiments on eight datasets with mainstream self-supervised pretraining models and supervised models. The results demonstrated that GPHT surpasses the baseline models across various fine-tuning and zero/few-shot learning settings in the traditional long-term forecasting task.
  arXiv  Detail & Related papers  (2024-02-26T11:54:54Z)
• Cumulative Distribution Function based General Temporal Point Processes [49.758080415846884]
  CuFun model represents a novel approach to TPPs that revolves around the Cumulative Distribution Function (CDF) Our approach addresses several critical issues inherent in traditional TPP modeling. Our contributions encompass the introduction of a pioneering CDF-based TPP model, the development of a methodology for incorporating past event information into future event prediction.
  arXiv  Detail & Related papers  (2024-02-01T07:21:30Z)
• TrackGPT -- A generative pre-trained transformer for cross-domain entity trajectory forecasting [0.0]
  We introduce TrackGPT, a Generative Pre-trained Transformer (GPT)-based model for entity trajectory forecasting. TrackGPT stands as a pioneering GPT model capable of producing accurate predictions across diverse entity time series datasets. We present benchmarks against state-of-the-art deep learning techniques, showing that TrackGPT's forecasting capability excels in terms of accuracy, reliability, and modularity.
  arXiv  Detail & Related papers  (2024-01-29T20:05:14Z)
• TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting [24.834846119163885]
  We propose a novel framework, TEMPO, that can effectively learn time series representations. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains.
  arXiv  Detail & Related papers  (2023-10-08T00:02:25Z)
• Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models [62.838689691468666]
  We propose Federated Black-Box Prompt Tuning (Fed-BBPT) to optimally harness each local dataset. Fed-BBPT capitalizes on a central server that aids local users in collaboratively training a prompt generator through regular aggregation. Relative to extensive fine-tuning, Fed-BBPT proficiently sidesteps memory challenges tied to PTM storage and fine-tuning on local machines.
  arXiv  Detail & Related papers  (2023-10-04T19:30:49Z)
• LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models [31.121714473817793]
  Foundation models have demonstrated a great ability to achieve general human-level intelligence far beyond traditional approaches. A significant shortcoming of most foundation models lies in their performance in specialized-domain and task-specific applications. We introduce LMFlow, which aims to simplify the domain- and task-aware finetuning of general foundation models.
  arXiv  Detail & Related papers  (2023-06-21T17:58:25Z)
• TSGM: A Flexible Framework for Generative Modeling of Synthetic Time Series [61.436361263605114]
  Time series data are often scarce or highly sensitive, which precludes the sharing of data between researchers and industrial organizations. We introduce Time Series Generative Modeling (TSGM), an open-source framework for the generative modeling of synthetic time series.
  arXiv  Detail & Related papers  (2023-05-19T10:11:21Z)
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models [152.29364079385635]
  As pre-trained models grow bigger, the fine-tuning process can be time-consuming and computationally expensive. We propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning and (ii) resource-efficient inference.
  arXiv  Detail & Related papers  (2021-10-30T03:29:47Z)
• Transformer Hawkes Process [79.16290557505211]
  We propose a Transformer Hawkes Process (THP) model, which leverages the self-attention mechanism to capture long-term dependencies. THP outperforms existing models in terms of both likelihood and event prediction accuracy by a notable margin. We provide a concrete example, where THP achieves improved prediction performance for learning multiple point processes when incorporating their relational information.
  arXiv  Detail & Related papers  (2020-02-21T13:48:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.