Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting

• URL: http://arxiv.org/abs/2605.27668v1
• Date: Tue, 26 May 2026 20:39:33 GMT
• Title: Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
• Abstract summary: We propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood.<n>Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models.
• Score: 10.121727570560305
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Probabilistic forecasting estimates the likelihood of uncertain future events. To improve LLM forecasting, existing methods typically learn from binary outcomes to output verbalized forecasts. However, while aggregated human forecasts contain rich information in both the crowd probability estimate and the degree of agreement among forecasters, how to utilize these signals remains underexplored. To address this, we propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood, using supervision from both binary outcomes and human forecasts. BBC models event likelihood $p \sim \text{Beta}(α, β)$ and outcome $y \sim \text{Bernoulli}(p)$, with the mean as the calibrated point forecast and the variance as the epistemic uncertainty. Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models fine-tuned specifically for forecasting, while remaining lightweight and having good generalization. We also show that the epistemic uncertainty captured by BBC is a more reliable predictor of forecasting error than verbalized confidence.

Related papers

• Latent-Regime Bias Auditing for Volatility Forecasting [0.44333156307871585]
  This paper proposes a model-agnostic audit framework for evaluating whether volatility forecasts remain reliable across latent market regimes.<n>We learn time-series representations of market-state windows, cluster them into regimes using only training information, and compare aggregate forecast behavior with regime-conditional bias, tail-underprediction, and underprediction-sensitive economic losses.
  arXiv  Detail & Related papers  (2026-08-03T02:09:39Z)
• Rigorous uncertainty quantification of probabilistic AI weather forecasts with conformal prediction [9.24882416591115]
  Probabilistic weather forecasting is undergoing rapid transformation with artificial intelligence (AI)<n>In traditional numerical weather prediction, computing power can limit how well ensemble forecasts approximate the unknown statistical distribution of future states.<n>Here we show that the statistical coverage of such models, the ultimate measure of calibration, can struggle, especially on extreme events.<n>We employ conformal prediction, a class of statistical methods that mathematically guarantees coverage under no distributional assumptions.
  arXiv  Detail & Related papers  (2026-06-17T22:41:20Z)
• Probabilistic bias adjustment of seasonal forecasts using generative machine learning: A case study of Arctic sea ice predictions [0.0]
  We develop a framework to generate large ensembles of bias adjusted seasonal predictions of Arctic sea ice.<n>We extend this framework to address the loss of fine-scale energy and the characteristic blurriness in predictions.<n>We show that the adjusted forecasts are better, more consistent with the observational distribution, and exhibit smaller errors than benchmark predictions.
  arXiv  Detail & Related papers  (2026-05-27T23:14:46Z)
• The Forecast Critic: Leveraging Large Language Models for Poor Forecast Identification [74.64864354503204]
  We propose The Forecast Critic, a system that leverages Large Language Models (LLMs) for automated forecast monitoring.<n>We evaluate the ability of LLMs to assess time series forecast quality.<n>We present three experiments, including on both synthetic and real-world forecasting data.
  arXiv  Detail & Related papers  (2025-12-12T21:59:53Z)
• Conditional Forecasts and Proper Scoring Rules for Reliable and Accurate Performative Predictions [1.1087735229999816]
  We show that conditioning forecasts on covariables that separate them from the outcome renders the target distribution forecast-invariant.<n>We identify two solutions: (i) in decision-theoretic settings, elicitation of correct and incentive-compatible forecasts is possible if forecasts are separating; (ii) scoring with unbiased estimates of the divergence between the forecast and the induced distribution of the target variable yields correct forecasts.<n>Our results expose fundamental limits of classical forecast evaluation and offer new tools for reliable and accurate forecasting in performative settings.
  arXiv  Detail & Related papers  (2025-10-24T10:59:21Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions.<n>We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• Hybrid Forecasting of Geopolitical Events [71.73737011120103]
  SAGE is a hybrid forecasting system that combines human and machine generated forecasts.<n>The system aggregates human and machine forecasts weighting both for propinquity and based on assessed skill.<n>We show that skilled forecasters who had access to machine-generated forecasts outperformed those who only viewed historical data.
  arXiv  Detail & Related papers  (2024-12-14T22:09:45Z)
• Calibrated Large Language Models for Binary Question Answering [49.1574468325115]
  A well-calibrated model should produce probabilities that accurately reflect the likelihood of its predictions being correct. We propose a novel approach that utilizes the inductive Venn--Abers predictor (IVAP) to calibrate the probabilities associated with the output tokens corresponding to the binary labels.
  arXiv  Detail & Related papers  (2024-07-01T09:31:03Z)
• Movement-Prediction-Adjusted Naive Forecast [6.935130578959931]
  The movement-prediction-adjusted naive forecast (MPANF) is designed to improve point forecasts beyond the naive baseline.<n> MPANF can serve as an effective second-stage method when reliable movement predictions are available.
  arXiv  Detail & Related papers  (2024-06-20T16:32:18Z)
• ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast [57.6987191099507]
  We introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast. We also introduce ExBooster, which captures the uncertainty in prediction outcomes by employing multiple random samples. Our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.
  arXiv  Detail & Related papers  (2024-02-02T10:34:13Z)
• Uncertainty estimation of pedestrian future trajectory using Bayesian approximation [137.00426219455116]
  Under dynamic traffic scenarios, planning based on deterministic predictions is not trustworthy. The authors propose to quantify uncertainty during forecasting using approximation which deterministic approaches fail to capture. The effect of dropout weights and long-term prediction on future state uncertainty has been studied.
  arXiv  Detail & Related papers  (2022-05-04T04:23:38Z)
• Evaluation of Machine Learning Techniques for Forecast Uncertainty Quantification [0.13999481573773068]
  Ensemble forecasting is, so far, the most successful approach to produce relevant forecasts along with an estimation of their uncertainty. Main limitations of ensemble forecasting are the high computational cost and the difficulty to capture and quantify different sources of uncertainty. In this work proof-of-concept model experiments are conducted to examine the performance of ANNs trained to predict a corrected state of the system and the state uncertainty using only a single deterministic forecast as input.
  arXiv  Detail & Related papers  (2021-11-29T16:52:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.