A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out

• URL: http://arxiv.org/abs/2609.10357v1
• Date: Wed, 09 Sep 2026 15:52:55 GMT
• Title: A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out
• Abstract summary: Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining.<n>We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains.<n>Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining. The obvious remedy is a hold-out that postdates the models. We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains, every observation published after the last model was released, and every dataset rebuildable without an API key. Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested. We then ask what separates the wins from the losses, and report a negative result: the two intrinsic properties one would reach for -- seasonal strength and spectral entropy, measured on the input window -- do not account for the pattern, and seasonal strength is if anything negatively associated with the advantage. What does track it is corpus familiarity. Our largest gain (28% lower MASE than the best classical method, on weekly Wikipedia pageviews) falls on Wikipedia pageviews, the domain TimesFM's authors describe as the bulk of its pretraining corpus, at the same granularities and differing only in time window. Within the pretrained family, where every model forecasts identical series so that series difficulty cancels, the TimesFM family outranks the Chronos family by -0.53 ranks on Wikipedia against -0.09 everywhere else (1,500 vs. 754 series, Mann-Whitney p < 1e-5). We conclude that a temporal hold-out removes memorisation of a window but not familiarity with a domain, that benchmarks therefore need domain hold-outs stated relative to disclosed corpora, and that the practitioner's question is less which model is better than whether their domain is one the model was raised on.

Related papers

• A Fragility Spectrum for Recursive Language-Model Training [9.563027159959587]
  We show that different models behave very differently under the same process.<n>The unique 4-gram outcome after five generations ranges from 0.187 to 0.940 across checkpoints.<n>Changing the composition of the shared pool or mixing in human text keeps the Spearman correlation of the ordering at 0.91--0.97.
  arXiv  Detail & Related papers  (2026-09-10T06:56:22Z)
• A Calibration Audit of Confidence in Feed-Forward 3D Reconstruction [45.496856835431885]
  3D reconstruction models emit per-pixel confidence that downstream systems read as a reliability signal.<n>We score the confidence on four properties, how well it ranks error, whether its level is right on average, whether it holds across the confidence range, and whether its intervals cover the truth.<n>The confidence ranks error well, but the predicted uncertainty is too low when it is read under conditions that are not exactly those of training.
  arXiv  Detail & Related papers  (2026-08-30T10:21:34Z)
• Evaluating Generative Time-Series Models on Data with Point Masses [2.5536942228520814]
  We report what happens when such data is evaluated carefully.<n>The standard rolling-origin protocol can score a model on a window whose atom structure bears no resemblance to the dataset.<n>An autoregressive hurdle beats a conditional flow on five of six datasets by up to a factor of $153$.
  arXiv  Detail & Related papers  (2026-08-10T14:57:36Z)
• When Do Foundation Models Pay Off? A Break-Even Analysis of Pretrained Time Series Forecasters [0.0]
  Foundation models outperform classical methods at every evaluated training fraction on 15 of 30 datasets.<n>One robust deployment rule requires no model training: if n_train 700 and seasonality is non-negligible, use FM zero-shot and skip fine-tuning.
  arXiv  Detail & Related papers  (2026-07-06T10:48:11Z)
• Nous: A Predictive World Model for Long-Term Agent Memory [1.5229257192293202]
  Nous is a novel agent memory architecture grounded in the principle that knowledge is prediction, not storage.<n>Each incoming observation is scored by its information-theoretic surprise S = -log2 P(obs | D)<n>The primary stored artifact is the delta, a record of the shift from prior to posterior belief, rather than the fact itself.
  arXiv  Detail & Related papers  (2026-06-20T13:07:00Z)
• GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation [90.53485251837235]
  Time series foundation models excel in zero-shot forecasting, handling diverse tasks without explicit training. GIFT-Eval is a pioneering benchmark aimed at promoting evaluation across diverse datasets. GIFT-Eval encompasses 23 datasets over 144,000 time series and 177 million data points.
  arXiv  Detail & Related papers  (2024-10-14T11:29:38Z)
• Paloma: A Benchmark for Evaluating Language Model Fit [112.481957296585]
  Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.<n>We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains.
  arXiv  Detail & Related papers  (2023-12-16T19:12:45Z)
• LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised Time Series Anomaly Detection [49.52429991848581]
  We propose a Light and Anti-overfitting Retraining Approach (LARA) for deep variational auto-encoder based time series anomaly detection methods (VAEs) This work aims to make three novel contributions: 1) the retraining process is formulated as a convex problem and can converge at a fast rate as well as prevent overfitting; 2) designing a ruminate block, which leverages the historical data without the need to store them; and 3) mathematically proving that when fine-tuning the latent vector and reconstructed data, the linear formations can achieve the least adjusting errors between the ground truths and the fine-tuned ones.
  arXiv  Detail & Related papers  (2023-10-09T12:36:16Z)
• Unifying Language Learning Paradigms [96.35981503087567]
  We present a unified framework for pre-training models that are universally effective across datasets and setups. We show how different pre-training objectives can be cast as one another and how interpolating between different objectives can be effective. Our model also achieve strong results at in-context learning, outperforming 175B GPT-3 on zero-shot SuperGLUE and tripling the performance of T5-XXL on one-shot summarization.
  arXiv  Detail & Related papers  (2022-05-10T19:32:20Z)
• Forget Less, Count Better: A Domain-Incremental Self-Distillation Learning Benchmark for Lifelong Crowd Counting [51.44987756859706]
  Off-the-shelf methods have some drawbacks to handle multiple domains. Lifelong Crowd Counting aims at alleviating the catastrophic forgetting and improving the generalization ability.
  arXiv  Detail & Related papers  (2022-05-06T15:37:56Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.