Scaling Point-in-Time Language Models

• URL: http://arxiv.org/abs/2607.11889v2
• Date: Fri, 17 Jul 2026 09:32:58 GMT
• Title: Scaling Point-in-Time Language Models
• Abstract summary: Point-in-time language models are trained exclusively on text available up to each calendar date.<n>We show that this performance gap can be substantially narrowed through scale.<n>We release the complete pipeline--including dataset construction--to enable point-in-time language modeling.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences. Point-in-time language models--trained exclusively on text available up to each calendar date--eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts. We show that this performance gap can be substantially narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of monthly model checkpoints spanning 2013-2024. Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (e.g., Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks. Instruction fine-tuning via LoRA further improves downstream usability. We release the complete pipeline--including dataset construction, training infrastructure, and evaluation code--to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity.

Related papers

• LiFT: Does Instruction Fine-Tuning Improve In-Context Learning for Longitudinal Modelling by Large Language Models? [15.417461558991832]
  Longitudinal NLP tasks require reasoning over temporally ordered text to detect persistence and change in human behavior and opinions.<n>We introduce LiFT, a longitudinal instruction fine-tuning framework that unifies diverse longitudinal modeling tasks under a shared instruction schema.
  arXiv  Detail & Related papers  (2026-03-25T20:54:12Z)
• BEDTime: A Unified Benchmark for Automatically Describing Time Series [8.466823017204641]
  We argue that successful multi-modal models should be able to recognize, differentiate, and generate language descriptions of time series.<n>We then create BEDTime, the first benchmark dataset to assess models on each task.<n>Using BEDTime, we evaluate 13 state-of-the-art models, and find that dedicated time series foundation models severely underperform.
  arXiv  Detail & Related papers  (2025-09-05T16:18:20Z)
• PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics [5.645098175233682]
  We contribute a novel type of benchmark evaluating inductive reasoning capabilities of Large Language Models (LLMs)<n>We present a fully automated pipeline that generates problems with controllable difficulty, enabling evaluation of reasoning models.<n>Experiments reveal a substantial performance gap between models that leverage test-time compute or LCoT (long chain-of-thought) reasoning and those that do not.
  arXiv  Detail & Related papers  (2025-05-29T05:51:16Z)
• DATETIME: A new benchmark to measure LLM translation and reasoning capabilities [0.0]
  This paper introduces DATETIME, a new high-quality benchmark designed to evaluate the translation and reasoning abilities of a Large Language Model (LLM) on datetimes.<n>Datetimes are intuitive and straightforward for humans to process but present significant challenges for LLMs.<n>Our experiments show that state-of-the-art models exhibit significant difficulty with tasks involving reasoning on datetimes.
  arXiv  Detail & Related papers  (2025-04-22T17:52:04Z)
• Pretraining Language Models for Diachronic Linguistic Change Discovery [8.203894221271302]
  We show that efficient pretraining techniques can produce useful models over corpora too large for easy manual inspection.<n>We employ a novel date-attribution pipeline in order to obtain a temporally-segmented dataset of five 10-million-word slices.<n>We find that the pretrained models are faster to train than the finetuned baselines and that they better respect the historical divisions of our corpus.
  arXiv  Detail & Related papers  (2025-04-07T21:51:32Z)
• Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting [54.04430089029033]
  We present Lag-Llama, a general-purpose foundation model for time series forecasting based on a decoder-only transformer architecture. Lag-Llama is pretrained on a large corpus of diverse time series data from several domains, and demonstrates strong zero-shot generalization capabilities. When fine-tuned on relatively small fractions of such previously unseen datasets, Lag-Llama achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-10-12T12:29:32Z)
• Time-LLM: Time Series Forecasting by Reprogramming Large Language Models [110.20279343734548]
  Time series forecasting holds significant importance in many real-world dynamic systems. We present Time-LLM, a reprogramming framework to repurpose large language models for time series forecasting. Time-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models.
  arXiv  Detail & Related papers  (2023-10-03T01:31:25Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Efficient Nearest Neighbor Language Models [114.40866461741795]
  Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore. We show how to achieve up to a 6x speed-up in inference speed while retaining comparable performance.
  arXiv  Detail & Related papers  (2021-09-09T12:32:28Z)
• Pitfalls of Static Language Modelling [41.76918612574081]
  We show that state-of-the-art Transformer models perform worse in the realistic setup of predicting future utterances from beyond their training period. We argue that now is the right time to rethink our static language modelling evaluation protocol.
  arXiv  Detail & Related papers  (2021-02-03T09:01:49Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.