Time Waits for No One! Analysis and Challenges of Temporal Misalignment

• URL: http://arxiv.org/abs/2111.07408v1
• Date: Sun, 14 Nov 2021 18:29:19 GMT
• Title: Time Waits for No One! Analysis and Challenges of Temporal Misalignment
• Authors: Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, Noah A. Smith
• Abstract summary: We establish a suite of eight diverse tasks across different domains to quantify the effects of temporal misalignment. We find stronger effects of temporal misalignment on task performance than have been previously reported. Our findings motivate continued research to improve temporal robustness of NLP models.
• Score: 42.106972477571226
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.

Related papers

• Beyond Forecasting: Compositional Time Series Reasoning for End-to-End Task Execution [19.64976935450366]
  We introduce Compositional Time Series Reasoning, a new task of handling intricate multistep reasoning tasks from time series data. Specifically, this new task focuses on various question instances requiring structural and compositional reasoning abilities on time series data. We develop TS-Reasoner, a program-aided approach that utilizes large language model (LLM) to decompose a complex task into steps of programs.
  arXiv  Detail & Related papers  (2024-10-05T06:04:19Z)
• A Systematic Analysis on the Temporal Generalization of Language Models in Social Media [12.035331011654078]
  This paper focuses on temporal shifts in social media and, in particular, Twitter. We propose a unified evaluation scheme to assess the performance of language models (LMs) under temporal shift.
  arXiv  Detail & Related papers  (2024-05-15T05:41:06Z)
• Large Pre-trained time series models for cross-domain Time series analysis tasks [20.228846068418765]
  We propose a novel method of textitadaptive segmentation that automatically identifies optimal dataset-specific segmentation strategy during pre-training. This enables LPTM to perform similar to or better than domain-specific state-of-art model when fine-tuned to different downstream time-series analysis tasks and under zero-shot settings.
  arXiv  Detail & Related papers  (2023-11-19T20:16:16Z)
• UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecasting [59.11817101030137]
  This research advocates for a unified model paradigm that transcends domain boundaries. Learning an effective cross-domain model presents the following challenges. We propose UniTime for effective cross-domain time series learning.
  arXiv  Detail & Related papers  (2023-10-15T06:30:22Z)
• Effective Long-Context Scaling of Foundation Models [90.57254298730923]
  We present a series of long-context LLMs that support effective context windows of up to 32,768 tokens. Our models achieve consistent improvements on most regular tasks and significant improvements on long-context tasks over Llama 2.
  arXiv  Detail & Related papers  (2023-09-27T21:41:49Z)
• TeST: Test-time Self-Training under Distribution Shift [99.68465267994783]
  Test-Time Self-Training (TeST) is a technique that takes as input a model trained on some source data and a novel data distribution at test time. We find that models adapted using TeST significantly improve over baseline test-time adaptation algorithms.
  arXiv  Detail & Related papers  (2022-09-23T07:47:33Z)
• Temporal Effects on Pre-trained Models for Language Processing Tasks [9.819970078135343]
  We present a set of experiments with systems powered by large neural pretrained representations for English to demonstrate that em temporal model deterioration is not as big a concern. It is however the case that em temporal domain adaptation is beneficial, with better performance for a given time period possible when the system is trained on temporally more recent data.
  arXiv  Detail & Related papers  (2021-11-24T20:44:12Z)
• Learning Neural Models for Continuous-Time Sequences [0.0]
  We study the properties of continuous-time event sequences (CTES) and design robust yet scalable neural network-based models to overcome the aforementioned problems. In this work, we model the underlying generative distribution of events using marked temporal point processes (MTPP) to address a wide range of real-world problems.
  arXiv  Detail & Related papers  (2021-11-13T20:39:15Z)
• Temporally Correlated Task Scheduling for Sequence Learning [143.70523777803723]
  In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks. We introduce a learnable scheduler to sequence learning, which can adaptively select auxiliary tasks for training. Our method significantly improves the performance of simultaneous machine translation and stock trend forecasting.
  arXiv  Detail & Related papers  (2020-07-10T10:28:54Z)
• Clinical Risk Prediction with Temporal Probabilistic Asymmetric Multi-Task Learning [80.66108902283388]
  Multi-task learning methods should be used with caution for safety-critical applications, such as clinical risk prediction. Existing asymmetric multi-task learning methods tackle this negative transfer problem by performing knowledge transfer from tasks with low loss to tasks with high loss. We propose a novel temporal asymmetric multi-task learning model that performs knowledge transfer from certain tasks/timesteps to relevant uncertain tasks, based on feature-level uncertainty.
  arXiv  Detail & Related papers  (2020-06-23T06:01:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.