Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most

• URL: http://arxiv.org/abs/2605.22672v2
• Date: Fri, 22 May 2026 17:57:26 GMT
• Title: Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most
• Abstract summary: In inverse scaling, more capable models produce worse distributional forecasts on forecasting problems with superlinear growth and tail risk of regime change.<n>The pattern appears on ForecastBench-Sim, in forecasting synthetic SIR epidemics with a matched linear control, and replicates in real-world datasets on COVID-19, measles, housing markets, and hyperinflation.<n>This inverse scaling does not appear on single-threshold metrics common in LLM forecasting benchmarks, reversing the sign of the capability--accuracy relationship on identical outputs.
• Score: 11.4807141145402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We document inverse scaling in LLMs on forecasting problems whose underlying time series exhibit superlinear growth and tail risk of regime change, a structure common in finance and epidemiology. On these tasks, more capable models produce worse distributional forecasts. The pattern appears on ForecastBench-Sim (FBSim), a contamination-free, simulated-world benchmark we release, in forecasting synthetic SIR epidemics with a matched linear control, and replicates in real-world datasets on COVID-19, measles, housing markets, and hyperinflation. A per-quantile decomposition shows the failure concentrates at the upper tail, which more capable models shift upward to track aggressive extrapolations of growth, while the lower tail stays put. A within-family study of Llama-3.1 shows that both model scale and post-training independently contribute to this effect. Domain knowledge does not reliably rescue calibration. This inverse scaling does not appear on single-threshold metrics common in LLM forecasting benchmarks, reversing the sign of the capability--accuracy relationship on identical outputs. Single-threshold scoring at conventional cutoffs misses the upper-tail cost; tail-inclusive scoring reverses the sign of the capability--accuracy relationship on the same outputs. We recommend that LLM forecasting evaluations use continuous (and unbounded) measures of accuracy alongside bounded binary threshold metrics.

Related papers

• Hierarchical Exponential-Gaussian Mixtures for Watch-Time Distribution Prediction [74.05279072166697]
  Exponential-Gaussian Mixture Network (EGMN) models the full conditional WT distribution.<n>EGMN is vulnerable to variance collapse, component redundancy, and inactive components.<n>We propose a Hierarchical Exponential-Gaussian Mixture (HEGM) model that addresses these failure modes.
  arXiv  Detail & Related papers  (2026-08-24T15:06:18Z)
• Link Prediction or Perdition: the Seeds of Instability in Knowledge Graph Embeddings [2.1174215880331766]
  Embedding models (KGEMs) constitute the main link prediction approach to complete knowledge graphs.<n>Standard evaluation protocols emphasize rank-based metrics such as MRR or Hits@$K$, but usually overlook the influence of random seeds on result stability.<n>We find that high-performance models actually produce divergent predictions at the triple level and highly variable spaces.
  arXiv  Detail & Related papers  (2026-06-02T09:13:50Z)
• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• Nexus : An Agentic Framework for Time Series Forecasting [72.73790673303154]
  Time series forecasting requires reasoning with unstructured contextual data such as news or events.<n>We introduce Nexus, a multi-agent forecasting framework that decomposes prediction into specialized stages.<n>We show that current-generation LLMs possess substantially stronger intrinsic forecasting ability than previously recognized.
  arXiv  Detail & Related papers  (2026-05-14T05:12:13Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• Improving Machine Learning Performance with Synthetic Augmentation [0.0]
  We formalize synthetic augmentation as a modification of the effective training distribution.<n>Additional samples may reduce estimation error, but they may also shift the population objective.<n>We show that synthetic augmentation is beneficial only in variance-dominant regimes.<n>Rare-regime targeting can improve domain-specific metrics but may conflict with unconditional permutation inference.
  arXiv  Detail & Related papers  (2026-04-16T00:23:01Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• Beyond Confidence: The Rhythms of Reasoning in Generative Models [16.58205184223738]
  Large Language Models (LLMs) exhibit impressive capabilities yet suffer from sensitivity to slight input context variations, hampering reliability.<n>We introduce the Token Constraint Bound ($_mathrmTCB$), a novel metric that quantifies the maximum internal state an LLM can withstand before its dominant next-token prediction significantly changes.<n>Our experiments show $_mathrmTCB$ correlates with effective prompt engineering and uncovers critical prediction instabilities missed by perplexity during in-context learning and text generation.
  arXiv  Detail & Related papers  (2026-02-11T12:58:23Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Forecasting the U.S. Treasury Yield Curve: A Distributionally Robust Machine Learning Approach [0.12891210250935145]
  We study U.S. Treasury yield curve forecasting under distributional uncertainty.<n>Rather than minimizing average forecast error, the forecaster selects a decision rule that minimizes worst case expected loss.<n>We propose a distributionally robust ensemble forecasting framework that integrates factor models with high dimensional nonparametric machine learning models.
  arXiv  Detail & Related papers  (2026-01-08T05:26:43Z)
• Robust Probabilistic Load Forecasting for a Single Household: A Comparative Study from SARIMA to Transformers on the REFIT Dataset [0.0]
  This paper tackles the challenge using the volatile REFIT household dataset.<n>We first address this by conducting a rigorous comparative experiment to select a Seasonal Imputation method.<n>We then systematically evaluate a hierarchy of models, progressing from classical baselines to machine learning.<n>Our findings reveal that classical models fail to capture the data's non-linear, regime-switching behavior.
  arXiv  Detail & Related papers  (2025-11-30T12:05:18Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Enhancing Transformer-Based Foundation Models for Time Series Forecasting via Bagging, Boosting and Statistical Ensembles [7.787518725874443]
  Time series foundation models (TSFMs) have shown strong generalization and zero-shot capabilities for time series forecasting, anomaly detection, classification, and imputation.<n>This paper investigates a suite of statistical and ensemble-based enhancement techniques to improve robustness and accuracy.
  arXiv  Detail & Related papers  (2025-08-18T04:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.