Evaluating Generative Time-Series Models on Data with Point Masses

• URL: http://arxiv.org/abs/2608.09692v1
• Date: Mon, 10 Aug 2026 14:57:36 GMT
• Title: Evaluating Generative Time-Series Models on Data with Point Masses
• Abstract summary: We report what happens when such data is evaluated carefully.<n>The standard rolling-origin protocol can score a model on a window whose atom structure bears no resemblance to the dataset.<n>An autoregressive hurdle beats a conditional flow on five of six datasets by up to a factor of $153$.
• Score: 2.5536942228520814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many of the series that generative time-series models are benchmarked on place a large probability mass on a single value --- it does not rain, no ride is requested, no part is ordered. We report what happens when such data is evaluated carefully. First, the standard rolling-origin protocol can score a model on a window whose atom structure bears no resemblance to the dataset: on one benchmark the dataset is $42\%$ zeros and the evaluation windows are $13\%$, on another $47\%$ against $5\%$. This is not a cosmetic problem --- it reversed one of our own conclusions, turning the strongest occurrence model in our study into what looked like a cautionary tale. Second, we give a control in which CRPS is invariant \emph{by construction} while the temporal coupling is destroyed, which measures exactly how much that coupling contributes to a chosen statistic. Third, benchmarking seven models on a matched protocol over five seeds, an autoregressive hurdle beats a conditional flow on five of six datasets, by up to a factor of $153$, while the flow's own occurrence statistics vary by up to $62\%$ across training seeds and every baseline is deterministic. Finally, the model ordering is not the same under five different occurrence statistics, and the two that do not share a construction agree with each other least.

Related papers

• A Fragility Spectrum for Recursive Language-Model Training [9.563027159959587]
  We show that different models behave very differently under the same process.<n>The unique 4-gram outcome after five generations ranges from 0.187 to 0.940 across checkpoints.<n>Changing the composition of the shared pool or mixing in human text keeps the Spearman correlation of the ordering at 0.91--0.97.
  arXiv  Detail & Related papers  (2026-09-10T06:56:22Z)
• A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out [0.0]
  Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having seen the test set during pretraining.<n>We build one: thirteen forecasters -- four classical, three trained per dataset, six pretrained -- on seven groups drawn from five domains.<n>Under this protocol pretrained models win 5 of 7 groups, lose one to a Theta baseline, and on daily exchange rates are indistinguishable from a seasonal naive forecast, along with every other method tested.
  arXiv  Detail & Related papers  (2026-09-09T15:52:55Z)
• Distribution-First Population Simulation: Collapse, Calibration, and Recall in Non-WEIRD LLM Persona Modeling [0.0]
  Synthetic-population tools run every individual as an independent large language model (LLM) agent.<n>We show that this paradigm has a basic failure mode, and we set a distribution-first corrective against it.
  arXiv  Detail & Related papers  (2026-07-17T15:45:57Z)
• CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion [65.53153291902454]
  Collaborative Evaluation (CollabEval) is a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency.<n>We build a low-rank approximation of the score matrix, and use the reconstructed values to guarantee unbiased estimates of the true evaluation metric mean.<n>We find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.
  arXiv  Detail & Related papers  (2026-07-06T13:22:04Z)
• Self-Evolving Agents with Anytime-Valid Certificates [1.2691047660244335]
  We present textbfSEA, an architecture that confines self-modification to a small steering adapter and a versioned harness around a emphfrozen base model.<n>Five loop controllers compose published guarantees; because such gates can only emphselect among behaviors the frozen base already produces, five verifier-in-the-loop mechanisms supply the dense, grader-free signal the gates require.<n>Results are single-run on expensive evaluations; confirming run-to-run variance and adapting the per-task algorithm mix are future work.
  arXiv  Detail & Related papers  (2026-07-01T12:34:52Z)
• Towards Evaluating Data Priors for Tabular Foundation Models [74.75794086564035]
  We implement a unified interface for publicly available priors and priors constructed from real datasets.<n>We generate training tasks from each prior, train the same model architecture under a fixed training protocol, and evaluate the resulting models on shared downstream classification tasks.<n>Our results show that different priors favor different downstream behaviors, with some achieving stronger absolute performance and others exhibiting more consistent relative rankings across datasets.
  arXiv  Detail & Related papers  (2026-06-28T07:12:45Z)
• Nous: A Predictive World Model for Long-Term Agent Memory [1.5229257192293202]
  Nous is a novel agent memory architecture grounded in the principle that knowledge is prediction, not storage.<n>Each incoming observation is scored by its information-theoretic surprise S = -log2 P(obs | D)<n>The primary stored artifact is the delta, a record of the shift from prior to posterior belief, rather than the fact itself.
  arXiv  Detail & Related papers  (2026-06-20T13:07:00Z)
• The Surprising Universality of LLM Outputs: A Real-Time Verification Primitive [0.0]
  CPU-only scoring primitive runs at 2.6 microseconds per token.<n> token rank-frequency distributions converge to the same two- parameter Mandelbrot ranking distribution.<n>We derive a single-pass scoring primitive that composes with model log probabilities when available and degrades to a rank-only mode usable on closed APIs.
  arXiv  Detail & Related papers  (2026-04-28T13:35:31Z)
• QuitoBench: A High-Quality Open Time Series Forecasting Benchmark [32.21290355342465]
  Time series forecasting is critical across finance, healthcare, and cloud computing.<n>We introduce textscQuitoBench, a regime-balanced benchmark for time series forecasting.
  arXiv  Detail & Related papers  (2026-03-27T02:24:34Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• Beyond Invariance: Test-Time Label-Shift Adaptation for Distributions with "Spurious" Correlations [44.99833362998488]
  Changes in the data distribution at test time can have deleterious effects on the performance of predictive models. We propose a test-time label shift correction that adapts to changes in the joint distribution $p(y, z)$ using EM applied to unlabeled samples.
  arXiv  Detail & Related papers  (2022-11-28T18:52:33Z)
• Automatic sleep stage classification with deep residual networks in a mixed-cohort setting [63.52264764099532]
  We developed a novel deep neural network model to assess the generalizability of several large-scale cohorts. Overall classification accuracy improved with increasing fractions of training data.
  arXiv  Detail & Related papers  (2020-08-21T10:48:35Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.