Time Masking for Temporal Language Models

• URL: http://arxiv.org/abs/2110.06366v2
• Date: Thu, 14 Oct 2021 07:23:41 GMT
• Title: Time Masking for Temporal Language Models
• Authors: Guy D. Rosin, Ido Guy, Kira Radinsky
• Abstract summary: We propose a temporal contextual language model called TempoBERT, which uses time as an additional context of texts. Our technique is based on modifying texts with temporal information and performing time masking - specific masking for the supplementary time information.
• Score: 23.08079115356717
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Our world is constantly evolving, and so is the content on the web. Consequently, our languages, often said to mirror the world, are dynamic in nature. However, most current contextual language models are static and cannot adapt to changes over time. In this work, we propose a temporal contextual language model called TempoBERT, which uses time as an additional context of texts. Our technique is based on modifying texts with temporal information and performing time masking - specific masking for the supplementary time information. We leverage our approach for the tasks of semantic change detection and sentence time prediction, experimenting on diverse datasets in terms of time, size, genre, and language. Our extensive evaluation shows that both tasks benefit from exploiting time masking.

Related papers

• MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval [53.417646562344906]
  Video Moment Retrieval (VMR) aims to localize a specific temporal segment within an untrimmed long video given a natural language query. Existing methods often suffer from inadequate training annotations, i.e., the sentence typically matches with a fraction of the prominent video content in the foreground with limited wording diversity. This intrinsic modality imbalance leaves a considerable portion of visual information remaining unaligned with text. In this work, we take an MLLM as a video narrator to generate plausible textual descriptions of the video, thereby mitigating the modality imbalance and boosting the temporal localization.
  arXiv  Detail & Related papers  (2024-06-25T18:39:43Z)
• Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models [24.784375155633427]
  BiTimeBERT 2.0 is a novel language model pre-trained on a temporal news article collection. Each objective targets a unique aspect of temporal information. Results consistently demonstrate that BiTimeBERT 2.0 outperforms models like BERT and other existing pre-trained models.
  arXiv  Detail & Related papers  (2024-06-04T00:30:37Z)
• Time Machine GPT [15.661920010658626]
  Large language models (LLMs) are often trained on extensive, temporally indiscriminate text corpora. This approach is not aligned with the evolving nature of language. This paper presents a new approach: a series of point-in-time LLMs called Time Machine GPT (TiMaGPT)
  arXiv  Detail & Related papers  (2024-04-29T09:34:25Z)
• Temporal Validity Change Prediction [20.108317515225504]
  Existing benchmarking tasks require models to identify the temporal validity duration of a single statement. In many cases, additional contextual information, such as sentences in a story or posts on a social media profile, can be collected from the available text stream. We propose Temporal Validity Change Prediction, a natural language processing task benchmarking the capability of machine learning models to detect contextual statements that induce such change.
  arXiv  Detail & Related papers  (2024-01-01T14:58:53Z)
• Time is Encoded in the Weights of Finetuned Language Models [65.71926562424795]
  We present time vectors, a simple tool to customize language models to new time periods. Time vectors are created by finetuning a language model on data from a single time. This vector specifies a direction in weight space that, as our experiments show, improves performance on text from that time period.
  arXiv  Detail & Related papers  (2023-12-20T20:04:45Z)
• Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training [56.74440457571821]
  We analyze tasks covering syntax, semantics and reasoning, across 2M pre-training steps and five seeds. We identify critical learning phases across tasks and time, during which subspaces emerge, share information, and later disentangle to specialize. Our findings have implications for model interpretability, multi-task learning, and learning from limited data.
  arXiv  Detail & Related papers  (2023-10-25T09:09:55Z)
• Temporal Perceiving Video-Language Pre-training [112.1790287726804]
  This work introduces a novel text-video localization pre-text task to enable fine-grained temporal and semantic alignment. Specifically, text-video localization consists of moment retrieval, which predicts start and end boundaries in videos given the text description. Our method connects the fine-grained frame representations with the word representations and implicitly distinguishes representations of different instances in the single modality.
  arXiv  Detail & Related papers  (2023-01-18T12:15:47Z)
• Temporal Attention for Language Models [24.34396762188068]
  We extend the key component of the transformer architecture, i.e., the self-attention mechanism, and propose temporal attention. temporal attention can be applied to any transformer model and requires the input texts to be accompanied with their relevant time points. We leverage these representations for the task of semantic change detection. Our proposed model achieves state-of-the-art results on all the datasets.
  arXiv  Detail & Related papers  (2022-02-04T11:55:34Z)
• Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation [63.195935452646815]
  We propose a method to automatically generate a domain- and task-adaptive maskings of the given text for self-supervised pre-training. We present a novel reinforcement learning-based framework which learns the masking policy. We validate our Neural Mask Generator (NMG) on several question answering and text classification datasets.
  arXiv  Detail & Related papers  (2020-10-06T13:27:01Z)
• Enriching Word Embeddings with Temporal and Spatial Information [37.0220769789037]
  We present a model for learning word representation conditioned on time and location. We train our model on time- and location-stamped corpora, and show using both quantitative and qualitative evaluations that it can capture semantics across time and locations.
  arXiv  Detail & Related papers  (2020-10-02T03:15:03Z)
• Local-Global Video-Text Interactions for Temporal Grounding [77.5114709695216]
  This paper addresses the problem of text-to-video temporal grounding, which aims to identify the time interval in a video semantically relevant to a text query. We tackle this problem using a novel regression-based model that learns to extract a collection of mid-level features for semantic phrases in a text query. The proposed method effectively predicts the target time interval by exploiting contextual information from local to global.
  arXiv  Detail & Related papers  (2020-04-16T08:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.